Generative AI Google Brain

gemini-evaluation-evaltask-approach-bring-your-own-autorater-with-custom-metric

3 min read

Bring-Your-Own-Autorater using CustomMetric

Source notebook

Repo path: gemini/evaluation/evaltask_approach/bring_your_own_autorater_with_custom_metric.ipynb · Open on GitHub · intermediate

Evaluates Gemini prompts with a client-side CustomMetric and a BYO Gemini autorater.

Summary

This notebook teaches how to define a locally computed Vertex AI evaluation CustomMetric that calls a custom autorater. It builds a linguistic_acceptability metric using google-genai with JSON schema output from gemini-2.5-flash, then evaluates three prompt templates with EvalTask. It displays metric results, explanations, radar and bar plots, and experiment runs.

Key code patterns

Initialize Vertex AI

PROJECT_ID = "[your-project-id]"
LOCATION = "us-central1"
 
import vertexai
vertexai.init(project=PROJECT_ID, location=LOCATION)

Sets the project and region used by Vertex AI evaluation and Gemini calls.

BYO autorater with JSON schema

client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)
response = client.models.generate_content(
    model="gemini-2.5-flash",
    contents=metric_prompt,
    config=types.GenerateContentConfig(
        response_mime_type="application/json",
        response_schema={"type": "OBJECT", "properties": {"score": {"type": "NUMBER"}, "explanation": {"type": "STRING"}}},
    ),
)
return response.parsed

Forces the autorater to return structured score and explanation fields.

CustomMetric function

def linguistic_acceptability_fn(instance: dict) -> dict:
    prompt = instance["prompt"]
    response = instance["response"]
    metric_prompt = metric_prompt_template.format(prompt=prompt, response=response)
    response_dict = get_autorater_response(metric_prompt)
    return {
        "linguistic_acceptability": response_dict["score"],
        "explanation": response_dict["explanation"],
    }

Maps each evaluated row into a model-graded metric result whose score key matches the metric name.

Run EvalTask across prompts

metrics = ["rouge_l_sum", "fluency", "coherence", linguistic_acceptability_metric]
eval_task = EvalTask(dataset=eval_dataset, metrics=metrics, experiment=experiment_name)
 
for i, prompt_template in enumerate(prompt_templates):
    eval_result = eval_task.evaluate(
        model="gemini-2.5-flash",
        prompt_template=prompt_template,
    )

Compares several prompt templates using built-in metrics plus the custom autorater metric.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI, Vertex Gen AI Evaluation Service
  • SDKs / libraries: google-cloud-aiplatform[evaluation], google-genai, vertexai, pandas

When to use this

Use this pattern when you need Vertex AI EvalTask workflows but want a locally defined model-based metric powered by your own autorater.

Gotchas & caveats

  • PROJECT_ID must be set before calling vertexai.init.
  • The notebook installs updated packages and requires a runtime restart.
  • Colab authentication is needed when running in Colab.
  • CustomMetric instances are computed on the client side without using Vertex Gen AI Evaluation Service APIs.
  • The score field returned by the custom metric function must match the metric name.
  • Rate limiting for the autorater may need Eval SDK RateLimiter utilities based on capacity and QPS.

Best practices

  • Use response_mime_type and response_schema to get structured autorater output.
  • Return both a numeric metric score and an explanation from the custom metric.
  • Keep the custom metric score field name aligned with the CustomMetric name.
  • Compare multiple prompt templates under the same EvalTask dataset and metrics.
  • Use notebook_utils to display results, explanations, radar plots, bar plots, and experiment runs.

Graph View

Backlinks