Generative AI Google Brain

gemini-evaluation-evaluate-models-in-vertex-ai-studio-and-model-garden

3 min read

Use Gen AI Evaluation SDK to Evaluate Models in Vertex AI Studio, Model Garden, and Model Registry

Source notebook

Repo path: gemini/evaluation/evaluate_models_in_vertex_ai_studio_and_model_garden.ipynb · Open on GitHub · intermediate

Evaluates Gemini, Llama MaaS, Claude, and prompt templates with Vertex AI Gen AI Evaluation SDK.

Summary

This notebook teaches how to use the Vertex AI Python SDK for Gen AI Evaluation Service to evaluate first-party Gemini models, third-party models, and prompt templates. It loads a small OpenOrca sample, defines EvalTask objects with computation-based, pointwise, pairwise, and custom metrics, then runs evaluations against Gemini, Llama 3.1 MaaS, and Claude models. It also demonstrates prompt-template comparison, experiment logging, result visualization, explanations, and cleanup.

Key code patterns

Initialize Vertex AI

PROJECT_ID = "[your-project-id]"
LOCATION = "us-central1"
 
import vertexai
vertexai.init(project=PROJECT_ID, location=LOCATION)

Sets the project and region before running Vertex AI evaluation jobs.

Evaluate Gemini with EvalTask

rouge_eval_task = EvalTask(
    dataset=dataset,
    metrics=["rouge_l_sum"],
)
rouge_result = rouge_eval_task.evaluate(
    model="gemini-2.5-flash",
    prompt_template="# System_prompt\n{system_prompt} # Question\n{question}",
)

Shows the core evaluation flow using a dataset, metrics, model, and prompt template.

Custom pointwise metric

PointwiseMetric(
    metric="linguistic_acceptability",
    metric_prompt_template=PointwiseMetricPromptTemplate(
        criteria={"Proper Grammar": "..."},
        rating_rubric={"5": "Excellent", "1": "Poor"},
        input_variables=["prompt", "reference"],
    ),
)

Demonstrates how to define a model-based metric with criteria, rubric, and inputs.

Pairwise model comparison

PairwiseMetric(
    metric=PAIRWISE_METRIC_NAME,
    metric_prompt_template=MetricPromptTemplateExamples.get_prompt_template(PAIRWISE_METRIC_NAME),
    baseline_model=GenerativeModel("gemini-2.0-flash-lite"),
)

Compares a candidate model against a baseline model using side-by-side metrics.

Evaluate external model function

def llama_model_fn(prompt: str) -> str:
    response = client.chat.completions.create(
        model=MODEL_ID,
        messages=[{"role": "user", "content": prompt}],
    )
    return response.choices[0].message.content
 
llama_result = pointwise_eval_task.evaluate(model=llama_model_fn)

Uses a Python function adapter so EvalTask can evaluate a third-party endpoint or SDK client.

Compare prompt templates

for i, prompt_template in enumerate(prompt_templates):
    eval_result = summarization_eval_task.evaluate(
        prompt_template=prompt_template,
        model=GenerativeModel("gemini-2.0-flash", generation_config={"temperature": 0.3}),
        evaluation_service_qps=5,
    )

Runs the same task over multiple prompt templates to compare prompt quality.

Models & APIs used

  • Models: gemini-2.5-flash, gemini-2.0-flash-lite, meta/llama-3.1-8b-instruct-maas, claude-3-5-sonnet@20240620, gemini-2.0-flash
  • APIs / services: Vertex AI, Gen AI Evaluation Service, Vertex AI Model Garden, Vertex AI Model Registry
  • SDKs / libraries: google-cloud-aiplatform[evaluation], vertexai, datasets, anthropic[vertex], openai, pandas

When to use this

Use this pattern when selecting models, comparing prompts, or evaluating model responses with consistent Vertex AI Gen AI Evaluation metrics.

Gotchas & caveats

  • PROJECT_ID must be set before initializing Vertex AI.
  • The notebook was tested with Python 3.10.
  • Colab requires auth.authenticate_user().
  • Llama 3.1 MaaS is noted as supported only in us-central1.
  • Default credentials access tokens live for 1 hour by default and must be refreshed after expiration.
  • Claude model choices have region-specific availability in the notebook.
  • Prompt engineering evaluation recommends around 100 examples for high-quality aggregated metrics and statistical significance.
  • The cleanup cell deletes the configured experiment when delete_experiment is True.

Best practices

  • Use the same EvalTask configuration with fixed dataset and metrics when comparing model architectures.
  • Use prompt_template variables that match columns in the evaluation dataset.
  • Use model-based pointwise, pairwise, and computation-based metrics for different evaluation needs.
  • Inspect evaluation results and explanations with notebook_utils display helpers.
  • Log prompt-template evaluations to an experiment and visualize comparisons with radar and bar plots.
  • Customize evaluation_service_qps only with awareness of the documented quota guidance.

Graph View

Backlinks