Generative AI Google Brain

gemini-evaluation-evaltask-approach-intro-batch-evaluation

3 min read

Intro to Batch Evaluations with the Gemini API

Source notebook

Repo path: gemini/evaluation/evaltask_approach/intro_batch_evaluation.ipynb · Open on GitHub · intro

Runs asynchronous batch evaluation of Gemini responses with Vertex AI and Cloud Storage.

Summary

This notebook teaches how to run batch evaluations with the Gemini API in Vertex AI using Cloud Storage JSONL input. It builds a pointwise model-based fluency metric, uploads prompt/reference/response rows to GCS, submits an evaluateDataset long-running operation, polls until completion, then reads row-level and aggregate results from GCS.

Key code patterns

Initialize project and bucket

PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")
LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "us-central1")
BUCKET_URI = f"gs://{BUCKET_NAME}"
!gcloud storage buckets create {BUCKET_URI} --location={LOCATION}
vertexai.init(project=PROJECT_ID, location=LOCATION)

Sets the Google Cloud project, region, GCS output location, and Vertex AI client context.

Define pointwise metric

metrics = [{
    "pointwise_metric_spec": {
        "metric_prompt_template": "Evaluate the fluency of this sentence: {response}. Give score from 0 to 1. 0 - not fluent at all. 1 - very fluent."
    },
    "aggregation_metrics": ["AVERAGE", "MEDIAN"],
}]

Configures a model-based autorater metric and requests aggregate statistics.

Write JSONL input to GCS

eval_df = pd.DataFrame(eval_dict)
evaluation_file_uri = BUCKET_URI + "/pairwise_data.jsonl"
eval_df.to_json(evaluation_file_uri, orient="records", lines=True)

Creates the Cloud Storage JSONL dataset expected by the batch evaluation service.

Submit evaluateDataset request

request = {
    "dataset": {"gcs_source": {"uris": evaluation_file_uri}},
    "metrics": metrics,
    "output_config": {"gcs_destination": {"output_uri_prefix": BUCKET_URI}},
}
with open("pairwise_fluency_request.json", "w") as json_file:
    json.dump(request, json_file, indent=2)
operation = send_request("pairwise_fluency_request.json")

Packages dataset, metrics, and output destination into the REST batch evaluation request.

Poll and read results

while "done" not in get_operation(operation):
    time.sleep(30)
response_json = get_operation(operation)
output_uri = response_json["response"]["outputInfo"]["gcsOutputDirectory"]
evaluation_results = pd.read_json(output_uri + "/evaluation_results.jsonl", lines=True)
evaluation_results_agg = pd.read_json(output_uri + "/aggregation_results.jsonl", lines=True)

Handles the asynchronous operation and loads detailed and aggregate JSONL outputs.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI, Vertex AI Gen AI Eval service, Cloud Storage
  • SDKs / libraries: google-cloud-aiplatform[evaluation], vertexai, gcsfs, pandas

When to use this

Use this pattern when you need asynchronous, large-batch quality scoring of model responses with row-level and aggregate evaluation outputs.

Gotchas & caveats

  • The notebook requires an existing Google Cloud project and the Vertex AI API enabled.
  • Authentication is handled with gcloud auth print-access-token and curl rather than only the Vertex AI SDK.
  • The runnable path uses Cloud Storage JSONL input, while BigQuery is only mentioned as supported.
  • send_request builds a regional endpoint but hard-codes locations/us-central1 in the request path.
  • Batch jobs are asynchronous; the notebook polls every 30 seconds and notes Pub/Sub or Cloud Functions for production notification.

Best practices

  • Store each evaluation item as one JSON object per JSONL line.
  • Include prompt, response, and optional reference fields in the evaluation dataset.
  • Use aggregation metrics such as AVERAGE and MEDIAN for dataset-level summaries.
  • Write evaluation outputs to Cloud Storage and read evaluation_results.jsonl separately from aggregation_results.jsonl.
  • Use helper functions to parse nested JSON results into readable tabular output.

Graph View

Backlinks