Generative AI Google Brain

gemini-evaluation-evaltask-approach-evaluate-multimodal-task-image

3 min read

Evaluating multimodal task

Source notebook

Repo path: gemini/evaluation/evaltask_approach/evaluate_multimodal_task_image.ipynb · Open on GitHub · intermediate

Evaluates image-grounded car damage labels with Vertex AI EvalTask and a Gemini custom autorater.

Summary

This notebook teaches an experimental approach for evaluating a multimodal image task with the Vertex AI Python SDK for Gen AI Evaluation Service. It builds a car insurance assessment dataset with conversations, Cloud Storage image URIs, generated labels, and references, then defines a CustomMetric that uses gemini-2.5-flash as an autorater. The workflow runs EvalTask with exact_match and the custom coherence metric, displays summary and row-level results, and optionally deletes the experiment.

Key code patterns

Initialize Vertex AI

PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")
LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "us-central1")
vertexai.init(project=PROJECT_ID, location=LOCATION)

Sets project and region before using Vertex AI evaluation.

Structured Gemini autorater

client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)
response = client.models.generate_content(
    model="gemini-2.5-flash",
    contents=metric_prompt,
    config=types.GenerateContentConfig(
        response_mime_type="application/json",
        response_schema={"type": "OBJECT", "properties": {"score": {"type": "NUMBER"}, "explanation": {"type": "STRING"}}},
    ),
)
return response.parsed

Uses Gemini on Vertex AI with JSON schema output for metric scoring.

Multimodal metric prompt

image_file = types.Part.from_uri(
    file_uri=image_of_car_accident,
    mime_type="image/jpeg",
)
evaluation_prompt = [
    eval_instruction_template,
    "CONVERSATION: ", conversation,
    "IMAGE: ", image_file,
    "GENERATED RESPONSE: ", response,
]

Passes conversation text, a Cloud Storage image URI, and generated response into the autorater.

Custom EvalTask metric

custom_coherence_metric = CustomMetric(
    name="custom_coherence",
    metric_function=custom_coherence_fn,
)
metrics = ["exact_match", custom_coherence_metric]
eval_task = EvalTask(dataset=eval_dataset, metrics=metrics, experiment=experiment_name)
eval_result = eval_task.evaluate()

Combines a built-in exact match metric with a locally defined multimodal coherence metric.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI, Gen AI Evaluation Service, Cloud Storage
  • SDKs / libraries: google-cloud-aiplatform[evaluation], google-genai, vertexai, pandas

When to use this

Use this pattern to evaluate multimodal model outputs where text context, image evidence, and generated labels must be judged together.

Gotchas & caveats

  • The notebook requires installing google-cloud-aiplatform[evaluation] and google-genai, then restarting the runtime.
  • Colab users must authenticate with google.colab.auth.authenticate_user().
  • A Google Cloud project with the Vertex AI API enabled is required.
  • LOCATION defaults to us-central1 when GOOGLE_CLOUD_REGION is unset.
  • The notebook states the multimodal evaluation result depends on the autorater’s ability to handle multimodal inputs and criteria.
  • The image inputs are gs:// Cloud Storage URIs with mime_type image/jpeg.
  • The custom autorater disables blocking for several safety categories in the GenerateContentConfig.

Best practices

  • Use a structured response schema with score and explanation for autorater outputs.
  • Define a clear metric definition, criteria, rating rubric, and evaluation steps in the custom metric prompt.
  • Include both summary metrics and row-based metrics when reviewing evaluation results.
  • Inspect sampled explanations to validate the evaluation behavior.
  • Clean up the Vertex AI experiment when it is no longer needed.

Graph View

Backlinks