Generative AI Google Brain

gemini-evaluation-evaltask-approach-evaluating-prompts-at-scale-with-gemini-batch-prediction-api

3 min read

Evaluating prompts at scale with Gemini Batch Prediction API

Source notebook

Repo path: gemini/evaluation/evaltask_approach/evaluating_prompts_at_scale_with_gemini_batch_prediction_api.ipynb · Open on GitHub · intermediate

Evaluates Gemini image-classification prompts at scale with Batch Prediction and BigQuery.

Summary

The notebook evaluates a sports image-classification prompt by preparing ground truth in BigQuery, referencing images in Cloud Storage, and running Gemini Batch Prediction through Vertex AI. It first tests one image with the Google GenAI SDK, then writes per-image JSON requests to a BigQuery prompts table and submits a batch job with model gemini-2.5-flash. Results are parsed into BigQuery views, joined to ground truth for correctness, and intended for analysis in BigQuery and Looker Studio.

Key code patterns

Initialize clients

bq_client = bigquery.Client(project=PROJECT_ID)
storage_client = storage.Client()
vertexai.init(project=PROJECT_ID, location=LOCATION)
bpd.options.bigquery.project = PROJECT_ID
bpd.options.bigquery.location = LOCATION

Configures BigQuery, Cloud Storage, Vertex AI, and BigQuery DataFrames from the notebook constants.

One-image JSON check

client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)
response = client.models.generate_content(
    model=MODEL_ID,
    contents=[prompt, types.Part.from_uri(file_uri=f"{GCS_PREFIX}/{blob_name}", mime_type="image/jpeg")],
    config=types.GenerateContentConfig(response_mime_type="application/json"),
)
return response.text

Validates the multimodal prompt and JSON response format before launching the batch job.

Build batch requests

request = json.dumps({
    "contents": [{"role": "user", "parts": [
        {"text": prompt},
        {"fileData": {"mimeType": "image/jpeg", "fileUri": f"{GCS_PREFIX}/{image_uri}"}}
    ]}],
    "generationConfig": {"responseMimeType": "application/json"},
})
prompts_df.to_gbq(PROMPTS_TABLE, PROJECT_ID)

Creates one BigQuery prompt row per image with the request payload expected by batch prediction.

Submit batch job

text_generation_table = f"{TEXT_GENERATION_TABLE_PREFIX}_{evaluation_id}"
batch_job = BatchPredictionJob.submit(
    source_model=MODEL_ID,
    input_dataset=f"bq://{PROMPTS_TABLE}",
    output_uri_prefix=f"bq://{text_generation_table}",
)
while not batch_job.has_ended:
    time.sleep(10)
    batch_job.refresh()

Runs Gemini Batch Prediction from a BigQuery input table to a BigQuery output table and polls completion.

Parse and score

results_sql = """
SELECT evaluation_id, evaluation_ts, prompt_text, gcs_uri,
JSON_VALUE(JSON_VALUE(response, '$.candidates[0].content.parts[0].text'), '$.sport') AS label
FROM `{text_generation_table_prefix}_*`
"""
eval_sql = """
SELECT f.label, e.gcs_uri, f.label = e.label AS correct
FROM `{files_table}` f JOIN `{results_view}` e ON f.path = e.gcs_uri
"""

Extracts the model’s JSON sport label and joins it to ground truth for correctness analysis.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI, Gemini Batch Prediction API, BigQuery, Cloud Storage, Looker Studio
  • SDKs / libraries: google-genai, google-cloud-aiplatform, vertexai, google-cloud-bigquery, google-cloud-storage, bigframes, pandas, pandas-gbq

When to use this

Use this pattern to evaluate a Gemini prompt over many Cloud Storage images and analyze prediction quality in BigQuery.

Gotchas & caveats

  • PROJECT_ID is a placeholder and must be set before running the notebook.
  • The dependency cell is followed by an intentional Colab kernel restart.
  • Colab authentication only runs when the notebook detects google.colab.
  • LOCATION is set to us-central1 and is applied to Vertex AI and BigQuery DataFrames.
  • The request payload hard-codes image/jpeg for GCS image parts.
  • The BigQuery view creation cells are marked run only once and use exists_ok=False.
  • The results view reads tables matching the TEXT_GENERATION_TABLE_PREFIX wildcard.

Best practices

  • Test the prompt on one image with generate_content before launching batch prediction.
  • Request JSON output in both the prompt and generation config.
  • Store evaluation_ts, evaluation_id, prompt_text, and gcs_uri with every request row.
  • Load ground truth into BigQuery and join predictions to compute correctness.
  • Use BigQuery views to parse raw responses and expose an evaluation table.
  • Use Gemini Batch Prediction to evaluate many examples with one request.

Graph View

Backlinks