Generative AI Google Brain

gemini-evaluation-evaltask-approach-bring-your-own-computation-based-metric

2 min read

Bring your own computation-based CustomMetric

Source notebook

Repo path: gemini/evaluation/evaltask_approach/bring_your_own_computation_based_metric.ipynb · Open on GitHub · intermediate

Evaluates Gemini outputs with locally defined computation-based Vertex AI CustomMetric functions.

Summary

This notebook teaches how to use the Vertex AI Python SDK for Gen AI Evaluation Service with locally defined computation-based CustomMetric metrics. It installs the evaluation extras, authenticates in Colab, initializes Vertex AI, builds pandas evaluation datasets, defines word-count metrics, and runs EvalTask against gemini-2.5-flash or precomputed responses.

Key code patterns

Initialize Vertex AI

PROJECT_ID = "[your-project-id]"
LOCATION = "us-central1"
EXPERIMENT_NAME = "customize-metrics"
 
import vertexai
vertexai.init(project=PROJECT_ID, location=LOCATION)

Sets the Google Cloud project and location before running EvalTask experiments.

Define CustomMetric

def word_count(instance: dict[str, str]) -> dict[str, float]:
    response = instance["response"]
    score = len(response.split(" "))
    return {"word_count": score}
 
custom_word_count_metric = CustomMetric(
    name="word_count",
    metric_function=word_count,
)

Shows the required pattern for client-side computation metrics: accept an instance dict and return metric scores.

Evaluate Model With Mixed Metrics

result = EvalTask(
    dataset=eval_dataset,
    metrics=[
        custom_word_count_metric,
        MetricPromptTemplateExamples.Pointwise.VERBOSITY,
    ],
    experiment=EXPERIMENT_NAME,
).evaluate(model="gemini-2.5-flash")

Demonstrates mixing a local CustomMetric with a built-in prompt-template metric while evaluating a Gemini model.

Evaluate Without Inference

custom_metric_result = EvalTask(
    dataset=eval_dataset,
    metrics=[word_count_match_metric],
    experiment=EXPERIMENT_NAME,
).evaluate()

Uses existing response and reference columns to score outputs without calling a model.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI, Vertex Gen AI Evaluation Service
  • SDKs / libraries: google-cloud-aiplatform, vertexai, pandas

When to use this

Use this pattern when you need Vertex AI EvalTask runs with deterministic client-side metrics over Gemini outputs or existing responses.

Gotchas & caveats

  • Requires google-cloud-aiplatform[evaluation].
  • Colab runtime must restart after installing packages.
  • Colab authentication is required before using Google Cloud resources.
  • PROJECT_ID must be set and the Vertex AI API must be enabled.
  • Notebook uses LOCATION us-central1.
  • For best results, the notebook recommends at least 100 examples.
  • Custom metrics are computed on the client side without using Vertex Gen AI Evaluation Service APIs.

Best practices

  • Initialize vertexai with project and location before evaluation.
  • Use pandas DataFrame columns such as prompt, response, and reference for EvalTask datasets.
  • Return CustomMetric results as a dictionary from metric name to numeric score.
  • Use precomputed response and reference columns when evaluating without inference.
  • Display evaluation results with notebook_utils.display_eval_result.

Graph View

Backlinks