Generative AI Google Brain

gemini-evaluation-evaltask-approach-evaluate-translation

2 min read

Evaluate a Translation Model

Source notebook

Repo path: gemini/evaluation/evaltask_approach/evaluate_translation.ipynb · Open on GitHub · intro

Evaluates stored translation responses with Vertex AI EvalTask using BLEU, COMET, and MetricX.

Summary

This notebook teaches how to evaluate translation quality with the Vertex AI Python SDK for Gen AI Evaluation Service. It installs the evaluation extra, authenticates and initializes Vertex AI, builds a pandas dataset with source, response, and reference columns, runs an EvalTask with BLEU, COMET, and MetricX, displays summary and row metrics, then deletes the created ExperimentRun.

Key code patterns

Initialize Vertex AI

PROJECT_ID = "[your-project-id]"
LOCATION = "us-central1"
EXPERIMENT_NAME = "my-eval-task-experiment"
 
import vertexai
vertexai.init(project=PROJECT_ID, location=LOCATION)

Sets the project, region, and experiment context before running evaluation.

Configure translation metrics

from vertexai.evaluation.metrics import pointwise_metric
 
metrics = [
    "bleu",
    pointwise_metric.Comet(version="COMET_22_SRC_REF"),
    pointwise_metric.MetricX(version="METRICX_24_SRC"),
]

Combines BLEU with reference-based COMET and reference-free MetricX for translation scoring.

Build evaluation dataset

eval_dataset = pd.DataFrame({
    "source": sources,
    "response": responses,
    "reference": references,
})

Uses the required source, response, and reference fields for translation evaluation.

Run EvalTask

eval_task = evaluation.EvalTask(
    dataset=eval_dataset,
    metrics=metrics,
    experiment=EXPERIMENT_NAME,
)
eval_result = eval_task.evaluate()

Runs evaluation over stored model responses without calling a generation model.

Clean up ExperimentRun

from google.cloud import aiplatform
 
aiplatform.ExperimentRun(
    run_name=eval_result.metadata["experiment_run"],
    experiment=eval_result.metadata["experiment"],
).delete()

Removes the ExperimentRun created by the evaluation task.

Models & APIs used

  • APIs / services: Vertex AI, Gen AI Evaluation Service
  • SDKs / libraries: google-cloud-aiplatform[evaluation], vertexai, pandas

When to use this

Use this pattern when you already have translation outputs and need repeatable Vertex AI evaluation with BLEU, COMET, and MetricX.

Gotchas & caveats

  • The notebook requires setting PROJECT_ID before initialization.
  • After installing google-cloud-aiplatform[evaluation], the Jupyter runtime may need a restart.
  • Colab authentication is only run when google.colab is present in sys.modules.
  • The notebook initializes Vertex AI in us-central1.
  • The cleanup step deletes the ExperimentRun referenced in eval_result.metadata.

Best practices

  • Install google-cloud-aiplatform with the evaluation extra before importing evaluation APIs.
  • Initialize vertexai with an explicit project and location.
  • Use a structured pandas DataFrame with source, response, and reference columns.
  • Inspect both summary metrics and row-based metrics from EvalResult.
  • Delete the ExperimentRun created by evaluation when finished.

Graph View

Backlinks