Generative AI Google Brain

gemini-evaluation-model-migration-with-gen-ai-eval

3 min read

Migrating Foundation Models: A Practical Guide with Gen AI Evaluation Serivce

Source notebook

Repo path: gemini/evaluation/model_migration_with_gen_ai_eval.ipynb · Open on GitHub · advanced

Compares model migration candidates with Vertex AI Gen AI Evaluation and prompt optimization workflows.

Summary

This notebook teaches how to use the Vertex AI SDK for Gen AI Evaluation Service to generate rubrics, run inference, compare candidate model outputs, and visualize results. It demonstrates comparing Gemini 2.0 Flash and Gemini 2.5 Flash, evaluating a third-party OpenAI model via LiteLLM, running asynchronous batch evaluation to Cloud Storage, and optimizing prompts with VAPO before reading evaluation results.

Key code patterns

Create Vertex AI client

from vertexai import Client, types
client = Client(project=PROJECT_ID, location=LOCATION)

Initializes the Vertex AI SDK client used for evaluation and prompt optimization.

Generate adaptive rubrics

data_with_rubrics = client.evals.generate_rubrics(
    src=prompts_df,
    rubric_group_name="general_quality_rubrics",
    predefined_spec_name=types.RubricMetric.GENERAL_QUALITY,
)

Creates rubric data from prompts using the predefined GENERAL_QUALITY rubric metric.

Run inference for candidates

candidate_1 = client.evals.run_inference(
    model="gemini-2.0-flash",
    src=data_with_rubrics,
    config={"generate_content_config": {"temperature": 1.6}}
)
candidate_2 = client.evals.run_inference(
    model="gemini-2.5-flash",
    src=data_with_rubrics,
)

Produces comparable evaluation datasets for two model migration candidates.

Compare model outputs

comparison_result = client.evals.evaluate(
    dataset=[candidate_1, candidate_2],
    metrics=[types.RubricMetric.GENERAL_QUALITY(
        rubric_group_name="general_quality_rubrics")]
)
comparison_result.show()

Evaluates multiple candidates in one run and renders the comparison report.

Batch evaluation to GCS

batch_eval_job = client.evals.batch_evaluate(
    dataset=inference_result_saved,
    metrics=[types.RubricMetric.FLUENCY, types.Metric(name="bleu")],
    dest=GCS_DEST_BUCKET
)

Uses a long-running operation for large-scale evaluation jobs with output stored in Cloud Storage.

Prompt optimization

prompt_optimizer_result = client.prompt_optimizer.optimize(
    method="vapo",
    config={
        "config_path": vapo_config_json_path,
        "service_account": SERVICE_ACCOUNT,
        "wait_for_completion": True
    }
)

Runs VAPO prompt optimization from a GCS-hosted config and waits for completion.

Models & APIs used

When to use this

Use this pattern when comparing foundation model migration candidates or prompt variants with rubric and metric-based Vertex AI evaluations.

Gotchas & caveats

  • Requires an existing Google Cloud project with the Vertex AI API enabled.
  • The tutorial uses billable Vertex AI components.
  • Colab authentication is needed when running in Google Colab.
  • Third-party model evaluation requires provider API keys such as OPENAI_API_KEY.
  • The notebook warns that setting API keys directly in code is insecure.
  • Batch evaluation and prompt optimization require a GCS destination bucket.
  • Prompt optimization config requires a project number-derived service account.
  • The SDK install requires google-cloud-aiplatform[evaluation]>=1.111.0.

Best practices

  • Use predefined rubric metrics such as GENERAL_QUALITY for structured model comparison.
  • Compare multiple candidates by passing a list of datasets to evaluate().
  • Use .show() on EvaluationDataset and EvaluationResult for in-notebook reports.
  • Use batch_evaluate() for large datasets or when immediate results are not required.
  • Use environment variables or secure storage for API keys instead of hardcoding them.
  • Re-evaluate after prompt optimization to quantify prompt changes.

Graph View

Backlinks