Generative AI Google Brain

gemini-evaluation-evaltask-approach-compare-generative-ai-models

2 min read

Compare Generative AI Models

Source notebook

Repo path: gemini/evaluation/evaltask_approach/compare_generative_ai_models.ipynb · Open on GitHub · intermediate

Evaluates Gemini models on summarization with Vertex AI Gen AI Evaluation Service.

Summary

This notebook teaches how to use the Vertex AI Python SDK for Gen AI Evaluation Service to compare generative AI models on an EvalTask. It builds a small summarization dataset, defines pointwise metrics, evaluates gemini-2.5-pro and gemini-2.5-flash with the same prompt template, and visualizes results. It also demonstrates a PairwiseMetric run with a baseline model and candidate model comparison.

Key code patterns

Initialize Vertex AI

PROJECT_ID = "[your-project-id]"
LOCATION = "us-central1"
 
import vertexai
vertexai.init(project=PROJECT_ID, location=LOCATION)

Sets the Google Cloud project and region before using Vertex AI evaluation APIs.

Create EvalTask

summarization_eval_task = EvalTask(
    dataset=eval_dataset,
    metrics=[
        MetricPromptTemplateExamples.Pointwise.TEXT_QUALITY,
        MetricPromptTemplateExamples.Pointwise.FLUENCY,
        MetricPromptTemplateExamples.Pointwise.SAFETY,
        MetricPromptTemplateExamples.Pointwise.VERBOSITY,
    ],
    experiment=EXPERIMENT_NAME,
)

Defines the dataset, pointwise metrics, and experiment used for controlled model comparison.

Evaluate Multiple Models

prompt_template = "{instruction}. Article: {context}. Summary:"
 
for model_name, model in models.items():
    eval_result = summarization_eval_task.evaluate(
        model=model,
        prompt_template=prompt_template,
    )
    eval_results.append((f"Model {model_name}", eval_result))

Runs the same evaluation task and prompt template across models to isolate model impact.

Pairwise Evaluation

PairwiseMetric(
    metric="pairwise_text_quality",
    metric_prompt_template=MetricPromptTemplateExamples.get_prompt_template(metric_name),
    baseline_model="gemini-2.5-flash",
)

Shows how to configure a side-by-side model comparison with a baseline model.

Models & APIs used

  • Models: gemini-2.5-pro, gemini-2.5-flash
  • APIs / services: Vertex AI, Gen AI Evaluation Service
  • SDKs / libraries: vertexai, google-cloud-aiplatform[evaluation], pandas, plotly

When to use this

Use this pattern when selecting between Gemini models for a specific task using repeatable evaluation metrics.

Gotchas & caveats

  • The notebook requires setting PROJECT_ID before running Vertex AI initialization.
  • LOCATION is set to us-central1 in the notebook.
  • Colab authentication is only run when google.colab is detected.
  • The notebook installs google-cloud-aiplatform[evaluation] and requires a runtime restart.
  • The pairwise example sets both baseline_model and candidate model to gemini-2.5-flash in the shown code.

Best practices

  • Use the same dataset and prompt template across model runs for controlled comparison.
  • Define an EvalTask with explicit metrics before running evaluations.
  • Store runs under an experiment name for later display and comparison.
  • Inspect metric explanations, not only summary scores.
  • Visualize multiple metrics with a radar plot when comparing models.

Graph View

Backlinks