Generative AI Google Brain

gemini-evaluation-evaluate-videos-with-predefined-gecko

2 min read

Evaluate videos with predefined Gecko

Source notebook

Repo path: gemini/evaluation/evaluate_videos_with_predefined_gecko.ipynb · Open on GitHub · intermediate

Evaluates generated videos with Vertex AI Gecko text-to-video rubrics.

Summary

This notebook demonstrates how to use the Vertex AI evaluation service to run predefined Gecko evaluation for video generation outputs. It builds a pandas dataset of prompts and Cloud Storage video responses, generates prompt-specific Gecko rubrics, then evaluates each video response with the GECKO_TEXT2VIDEO metric.

Key code patterns

Initialize Vertex AI client

from vertexai import Client, types
 
PROJECT_ID = ""
LOCATION = "us-central1"
client = Client(project=PROJECT_ID, location=LOCATION)

Creates the regional Vertex AI client used for rubric generation and evaluation.

Build video evaluation dataset

eval_dataset = pd.DataFrame({
    "prompt": prompts,
    "response": responses,
})

Pairs text prompts with model responses that reference video/mp4 files in Cloud Storage.

Generate Gecko rubrics

data_with_rubrics = client.evals.generate_rubrics(
    src=eval_dataset,
    rubric_group_name="gecko_video_rubrics",
    predefined_spec_name=types.RubricMetric.GECKO_TEXT2VIDEO,
)

Creates prompt-specific Gecko rubrics before scoring video outputs.

Evaluate with predefined metric

eval_result = client.evals.evaluate(
    dataset=data_with_rubrics,
    metrics=[types.RubricMetric.GECKO_TEXT2VIDEO],
)
 
eval_result.show()

Runs the predefined Gecko text-to-video metric against the rubric-enriched dataset.

Models & APIs used

  • APIs / services: Vertex AI, Cloud Storage
  • SDKs / libraries: google-cloud-aiplatform[evaluation], vertexai, pandas

When to use this

Use this pattern when evaluating whether generated videos match their text prompts with prompt-specific Gecko rubrics.

Gotchas & caveats

  • Requires an existing Google Cloud project with the Vertex AI API enabled.
  • Uses billable Vertex AI components.
  • Colab requires explicit user authentication.
  • The notebook installs google-cloud-aiplatform[evaluation]>=1.122.0.
  • Video responses are referenced as gs:// Cloud Storage URIs with video/mp4 MIME type.

Best practices

  • Generate rubrics from the user prompts before evaluating responses.
  • Use similar counterexample prompts to demonstrate high-quality and low-quality response differences.
  • Inspect generated questions and validator reliability when analyzing quality.
  • Manually add questions when needed for an application.

Graph View

Backlinks