Generative AI Google Brain

tools-llmevalkit-prompt-management-tutorial

3 min read

Tutorial for Running Prompt Management and Evaluation

Source notebook

Repo path: tools/llmevalkit/prompt-management-tutorial.ipynb · Open on GitHub · intermediate

Runs prompt management, MathVista evaluation, and Vertex AI prompt optimization in a Streamlit app.

Summary

This notebook sets up the llmevalkit Streamlit app, configures Google Cloud auth, Cloud Storage, and IAM, then walks through creating and versioning a multimodal Gemini prompt. It demonstrates uploading a MathVista JSONL dataset, generating baseline responses, running model-based evaluation, launching Vertex AI prompt optimization, saving a better prompt version, and comparing records in a leaderboard.

Diagrams

image.gif — source

prompt_optimization_result.png — source

welcome_page.png — source

Key code patterns

Copy sample dataset

! gcloud storage cp gs://github-repo/prompts/prompt_optimizer/mathvista_dataset/mathvista_input.jsonl mathvista_input.jsonl

Uses Cloud Storage to retrieve the MathVista JSONL evaluation input.

Set project auth

PROJECT_ID = "[your-project-id]"
LOCATION = "[your-project-region]"
! gcloud auth application-default login
! gcloud config set project {PROJECT_ID}

Establishes Application Default Credentials and selects the Google Cloud project.

Create evaluation bucket

BUCKET_NAME = "[your-bucket-name]"
BUCKET_URI = f"gs://{BUCKET_NAME}"
! gcloud storage buckets create {BUCKET_URI} --location {LOCATION}

Creates a Cloud Storage bucket used by the prompt management and dataset workflow.

Grant service account roles

PROJECT_NUMBER = !gcloud projects describe {PROJECT_ID} --format="get(projectNumber)"
SERVICE_ACCOUNT = f"{PROJECT_NUMBER[0]}-compute@developer.gserviceaccount.com"
for role in ['aiplatform.user', 'storage.objectAdmin']:
    ! gcloud projects add-iam-policy-binding {PROJECT_ID} \
      --member=serviceAccount:{SERVICE_ACCOUNT} \
      --role=roles/{role} --condition=None

Gives the compute service account Vertex AI and Cloud Storage permissions needed by the app.

Run Streamlit app

! cd generative-ai/llmevalkit && streamlit run index.py & npx localtunnel --port 8501

Starts the local prompt management UI and exposes it through localtunnel.

Multimodal prompt template

Problem: {{query}}
Image: {{image}} @@@image/jpeg
Answer: {{target}}

Shows the prompt format for evaluating questions that include text, images, and ground truth answers.

Models & APIs used

  • Models: gemini-2.0-flash-001
  • APIs / services: Vertex AI, Cloud Storage, Cloud Functions, Cloud Run, Cloud Resource Manager

When to use this

Use this pattern when building a prompt lifecycle workflow with versioning, dataset-backed evaluation, and Vertex AI prompt optimization.

Gotchas & caveats

  • Requires cloning the generative-ai repo and installing requirements.txt.
  • Colab requires auth.authenticate_user(); other environments use gcloud auth application-default login.
  • A Google Cloud project and enabled APIs are required before using Vertex AI resources.
  • The .env file must be copied from src/.env.example and filled with BUCKET_NAME, PROJECT_ID, and SERVICE_ACCOUNT.
  • The bucket can be existing or created with gcloud storage buckets create.
  • The compute service account needs roles/aiplatform.user and roles/storage.objectAdmin.
  • The Streamlit app is exposed through localtunnel, whose password is the external IP.
  • Prompt optimization can take about 20 minutes and progress is viewed in Vertex AI custom jobs.

Best practices

  • Version prompts each time changes are saved so iterations can be compared.
  • Test a prompt with sample input before saving and evaluating it.
  • Create datasets in Cloud Storage and upload CSV, JSON, or JSONL files for evaluation.
  • Run an initial evaluation before prompt changes to establish a baseline.
  • Use human-in-the-loop rating and automated metrics for evaluation.
  • Save evaluation results to prompt records to track performance over time.
  • Compare optimized prompt versions and select the highest scoring instruction.

Graph View

Backlinks