Generative AI Google Brain

gemini-evaluation-evaltask-approach-migration-guide-preview-to-ga-sdk

3 min read

Gen AI Evaluation Service SDK Preview-to-GA Migration Guide

Source notebook

Repo path: gemini/evaluation/evaltask_approach/migration_guide_preview_to_GA_sdk.ipynb · Open on GitHub · intermediate

Migrates Vertex AI Gen AI Evaluation SDK preview patterns to GA EvalTask, pointwise, and pairwise metrics.

Summary

This notebook teaches how to migrate from preview Vertex AI Python SDK evaluation APIs to the GA Gen AI Evaluation Service SDK. It demonstrates installing and initializing Vertex AI, replacing discontinued metrics with custom PointwiseMetric prompt templates, adapting datasets to the GA prompt/response schema, and running side-by-side evaluation with PairwiseMetric. The examples use SQuAD-style RAG question-answering data and display evaluation summaries and explanations.

Key code patterns

Initialize Vertex AI

PROJECT_ID = "[your-project-id]"
LOCATION = "us-central1"
EXPERIMENT = "eval-migration-ga"
 
import vertexai
vertexai.init(project=PROJECT_ID, location=LOCATION)

Sets project and region context before creating EvalTask jobs.

GA evaluation imports

from vertexai.evaluation import (
    EvalTask,
    MetricPromptTemplateExamples,
    PairwiseMetric,
    PointwiseMetric,
)

Uses the GA import path instead of vertexai.preview.evaluation for evaluation classes.

Custom pointwise metric

relevance = PointwiseMetric(
    metric="relevance",
    metric_prompt_template=relevance_prompt_template,
)
 
EvalTask(
    dataset=eval_dataset,
    metrics=[relevance, helpfulness, fulfillment],
    experiment=EXPERIMENT,
).evaluate()

Recreates discontinued metrics with explicit rubrics and evaluation prompt templates.

GA prompt schema

new_eval_dataset = pd.DataFrame({
    "prompt": [
        "Answer the question: " + question + " Context: " + context
        for question, context in zip(questions, retrieved_contexts)
    ],
    "response": generated_answers,
})

Converts instruction and context columns into the GA prompt/response input schema.

Pairwise SxS evaluation

EvalTask(
    dataset=eval_dataset,
    metrics=[PairwiseMetric(
        metric="pairwise_text_quality",
        metric_prompt_template=MetricPromptTemplateExamples.get_prompt_template("pairwise_text_quality"),
        baseline_model="gemini-2.5-flash",
    )],
    experiment=EXPERIMENT,
).evaluate(model="gemini-2.5-flash")

Replaces AutoSxS-style evaluation with PairwiseMetric in EvalTask.

Bring your own responses

EvalTask(
    dataset=eval_dataset,
    metrics=[
        MetricPromptTemplateExamples.Pairwise.VERBOSITY,
        MetricPromptTemplateExamples.Pairwise.SAFETY,
    ],
    experiment=EXPERIMENT,
).evaluate(
    prompt_template="Answer the question: {question} Context: {context}",
    evaluation_service_qps=5,
)

Evaluates saved candidate and baseline responses with pairwise metrics and a prompt template.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI, Gen AI Evaluation Service
  • SDKs / libraries: google-cloud-aiplatform[evaluation], vertexai, pandas

When to use this

Use this pattern when migrating preview Vertex AI Gen AI evaluation code to GA EvalTask APIs for RAG quality or SxS model comparison.

Gotchas & caveats

  • Python 3.9 is the tested notebook environment.
  • The notebook installs google-cloud-aiplatform[evaluation] and says to restart the runtime before continuing.
  • Colab authentication is required when running in Google Colab.
  • PROJECT_ID must be set or the notebook raises ValueError.
  • GA imports change from vertexai.preview.evaluation to vertexai.evaluation.
  • Several preview metrics are discontinued, including question_answering_helpfulness, question_answering_relevance, question_answering_correctness, summarization_helpfulness, summarization_verbosity, and fulfillment.
  • The notebook says google-cloud-aiplatform[rapid-evaluation]==1.62.0 can be pinned to keep old preview metric support.
  • GA MetricPromptTemplateExamples expect prompt plus response or baseline_model_response rather than fine-grained instruction and context inputs.

Best practices

  • Use MetricPromptTemplateExamples and adjust them for your use case instead of relying on removed black-box metrics.
  • Define custom PointwiseMetric or PairwiseMetric rubrics when discontinued metrics are still needed.
  • Use instruction_following instead of fulfillment and verbosity instead of summarization_verbosity.
  • Assemble instruction and context into a single prompt for GA metric templates.
  • Use PairwiseMetric in EvalTask for side-by-side evaluation of two models.
  • Use evaluation_service_qps to control evaluation service request rate in BYOR pairwise evaluation.

Graph View

Backlinks