Generative AI Google Brain

gemini-evaluation-evaluate-images-with-predefined-gecko

2 min read

Evaluate images with predefined Gecko

Source notebook

Repo path: gemini/evaluation/evaluate_images_with_predefined_gecko.ipynb · Open on GitHub · intermediate

Evaluates text-to-image outputs with Vertex AI predefined Gecko image rubrics.

Summary

The notebook teaches how to use the Vertex AI evaluation service to run predefined Gecko for image generation evaluation. It builds a pandas dataset of prompts and image responses stored as Cloud Storage URIs, generates Gecko text-to-image rubrics from the prompts, then evaluates the image responses with those rubrics.

Key code patterns

Initialize Vertex AI client

from vertexai import Client, types
 
PROJECT_ID = ""
LOCATION = "us-central1"
client = Client(project=PROJECT_ID, location=LOCATION)

Creates the Vertex AI client used for rubric generation and evaluation.

Build evaluation dataset

eval_dataset = pd.DataFrame({
    "prompt": prompts,
    "response": responses,
})

Pairs each text prompt with a model response containing image file_data.

Reference image responses

{
    "parts": [{
        "file_data": {
            "mime_type": "image/png",
            "file_uri": "gs://cloud-samples-data/generative-ai/evaluation/images/coffee.png",
        }
    }],
    "role": "model",
}

Formats image outputs as model-role responses with Cloud Storage file URIs.

Generate Gecko rubrics

data_with_rubrics = client.evals.generate_rubrics(
    src=eval_dataset,
    rubric_group_name="gecko_image_rubrics",
    predefined_spec_name=types.RubricMetric.GECKO_TEXT2IMAGE,
)

Uses the predefined Gecko text-to-image spec to generate prompt-specific rubrics.

Evaluate with rubrics

eval_result = client.evals.evaluate(
    dataset=data_with_rubrics,
    metrics=[types.RubricMetric.GECKO_TEXT2IMAGE],
)

Runs the generated rubrics against image responses and returns evaluation results.

Models & APIs used

  • APIs / services: Vertex AI, Cloud Storage
  • SDKs / libraries: google-cloud-aiplatform, vertexai, pandas

When to use this

Use this pattern to score generated images against text prompts with prompt-specific Gecko rubrics in Vertex AI.

Gotchas & caveats

  • Requires google-cloud-aiplatform[evaluation]>=1.122.0.
  • Requires an existing Google Cloud project with the Vertex AI API enabled.
  • Colab users must authenticate with google.colab.auth.authenticate_user().
  • The notebook sets LOCATION to us-central1.
  • The tutorial uses billable Vertex AI components.
  • Image responses must be provided in the expected parts/file_data structure with mime_type and file_uri.

Best practices

  • Generate rubrics from the user prompts before evaluating responses.
  • Use counterexample prompts with the same images to demonstrate high and low quality evaluations.
  • Inspect generated questions and validator reliability when analyzing quality.
  • Manually add questions when desired for an application.
  • Review Vertex AI pricing and estimate costs before running evaluation.

Graph View

Backlinks