Generative AI Google Brain

gemini-evaluation-quick-start-gen-ai-eval

2 min read

Getting Started: Quick Gen AI Evaluation

Source notebook

Repo path: gemini/evaluation/quick_start_gen_ai_eval.ipynb · Open on GitHub · intro

Evaluates gemini-2.5-flash responses with Vertex AI Gen AI Eval Service using a default quality rubric.

Summary

This notebook teaches the quickest workflow for evaluating a single generative model with the Vertex AI SDK for Gen AI Eval Service. It installs the evaluation-enabled Vertex AI SDK, authenticates in Colab, configures project and location, builds a small prompt DataFrame, runs inference with gemini-2.5-flash, then evaluates the generated responses using the default GENERAL_QUALITY adaptive rubric-based metric.

Key code patterns

Create Vertex AI client

from vertexai import Client, types
client = Client(project=PROJECT_ID, location=LOCATION)

Initializes the Vertex AI SDK client for evaluation workflows.

Build evaluation prompts

eval_df = pd.DataFrame({
    "prompt": [
        "Explain software 'technical debt' using a concise analogy of planting a garden.",
        "Write a Python function to find the nth Fibonacci number using recursion with memoization, but without using any imports."
    ]
})

Uses a pandas DataFrame as the evaluation source data.

Run model inference

eval_dataset = client.evals.run_inference(
    model="gemini-2.5-flash",
    src=eval_df,
)
eval_dataset.show()

Generates model responses from the prompt dataset before evaluation.

Evaluate generated responses

eval_result = client.evals.evaluate(dataset=eval_dataset)
eval_result.show()

Runs the default GENERAL_QUALITY adaptive rubric-based evaluation.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI
  • SDKs / libraries: vertexai, pandas

When to use this

Use this pattern for a quick first evaluation of one Gemini model on a small prompt dataset in Vertex AI.

Gotchas & caveats

  • Requires an existing Google Cloud project with the Vertex AI API enabled.
  • Uses billable Vertex AI components.
  • Colab users must authenticate with google.colab.auth.authenticate_user().
  • The notebook installs google-cloud-aiplatform[evaluation]>=1.111.0 with force reinstall.
  • Defaults to LOCATION us-central1 unless GOOGLE_CLOUD_REGION is set.
  • PROJECT_ID falls back to GOOGLE_CLOUD_PROJECT when not provided.

Best practices

  • Enable the Vertex AI API before using the SDK.
  • Use environment variables for PROJECT_ID and LOCATION defaults.
  • Generate responses with run_inference() before calling evaluate().
  • Use the SDK’s automatic handling of common data formats to avoid manual conversions.
  • Inspect intermediate and final results with show().

Graph View

Backlinks