Generative AI Google Brain

gemini-evaluation-evaltask-approach-customize-model-based-metrics

3 min read

Customize Model-based Metrics to Evaluate a Gen AI model

Source notebook

Repo path: gemini/evaluation/evaltask_approach/customize_model_based_metrics.ipynb · Open on GitHub · intermediate

Shows how to customize Vertex AI Gen AI Evaluation model-based metrics for Gemini summarization outputs.

Summary

The notebook teaches how to use the Vertex AI Python SDK for Gen AI Evaluation Service to define custom model-based metrics. It builds a small summarization evaluation dataset from XSum examples, then evaluates gemini-2.5-flash with templated pointwise and pairwise metrics. It also demonstrates fully custom prompt templates for pointwise and pairwise verbosity and reference-alignment evaluation.

Key code patterns

Initialize Vertex AI

PROJECT_ID = "[your-project-id]"
LOCATION = "us-central1"
EXPERIMENT_NAME = "customize-metrics"
 
import vertexai
vertexai.init(project=PROJECT_ID, location=LOCATION)

Sets project, region, and experiment context before running evaluation jobs.

Build evaluation dataset

eval_dataset = pd.DataFrame({
    "prompt": [instruction + item for item in context],
    "reference": reference,
})

Creates the required prompt and reference columns used by custom metric templates.

Templated pointwise metric

linguistic_acceptability = PointwiseMetric(
    metric="linguistic_acceptability",
    metric_prompt_template=PointwiseMetricPromptTemplate(
        criteria={"Proper Grammar": "..."},
        rating_rubric={"5": "Excellent", "1": "Poor"},
        input_variables=["prompt", "reference"],
    ),
)

Uses predefined template fields for criteria, rubric, and dataset variables.

Evaluate model with EvalTask

result = EvalTask(
    dataset=eval_dataset,
    metrics=[linguistic_acceptability],
    experiment=EXPERIMENT_NAME,
).evaluate(model="gemini-2.5-flash")

Runs the evaluation task against a Gemini model using the configured metrics.

Free-form metric prompt

PointwiseMetric(
    metric="free_form_pointwise_text_quality",
    metric_prompt_template=free_form_pointwise_metric_prompt,
)

Allows full control over evaluator instructions, criteria, rubric, and output fields.

Pairwise baseline comparison

PairwiseMetric(
    metric="free_form_pairwise_verbosity_reference_alignment",
    metric_prompt_template=free_form_pairwise_metric_prompt,
    baseline_model="gemini-2.5-flash",
)

Compares a model response against a baseline model response using pairwise choice labels.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI, Vertex Gen AI Evaluation Service
  • SDKs / libraries: vertexai, google-cloud-aiplatform[evaluation], pandas

When to use this

Use this pattern when you need custom evaluator criteria, rubrics, or pairwise comparisons for Gemini model outputs.

Gotchas & caveats

  • Requires google-cloud-aiplatform[evaluation] installation and runtime restart in Colab.
  • Requires Google Cloud authentication in Colab.
  • Requires PROJECT_ID to be set and Vertex AI API enabled.
  • The notebook uses LOCATION = us-central1.
  • The dataset example is small, while the notebook recommends at least 100 examples for best results.

Best practices

  • Initialize Vertex AI with an explicit project and location.
  • Use input_variables that match columns in the evaluation dataset.
  • Define clear criteria and rating rubrics for model-based metrics.
  • Use reference answers when evaluating summarization alignment.
  • Inspect reusable templates with MetricPromptTemplateExamples before customizing.
  • Display evaluation results with notebook_utils.display_eval_result.

Graph View

Backlinks