Generative AI Google Brain

gemini-evaluation-evaltask-approach-evaluate-videos-with-gecko

2 min read

Evaluate videos with Gecko

Source notebook

Repo path: gemini/evaluation/evaltask_approach/evaluate_videos_with_gecko.ipynb · Open on GitHub · advanced

Uses Vertex AI evaluation to run Gecko-style rubric generation and video validation.

Summary

The notebook teaches how to evaluate generated videos with Gecko using Vertex AI evaluation. It builds a prompt-derived rubric with multiple-choice QA records, validates each generated video against those questions, then computes per-row and aggregate scores. The workflow installs and initializes the Vertex AI SDK, defines parsing helpers and prompt templates, prepares prompt/video rows from Cloud Storage URIs, generates rubrics, and evaluates videos with an EvalTask.

Key code patterns

Initialize Vertex AI

PROJECT_ID = "your-project-id"
LOCATION = "us-central1"
 
import vertexai
vertexai.init(project=PROJECT_ID, location=LOCATION)

Sets the Google Cloud project and region before using Vertex AI evaluation.

Custom rubric parsing

rubric_generation_config = RubricGenerationConfig(
    prompt_template=RUBRIC_GENERATION_PROMPT,
    parsing_fn=parse_json_to_qa_records,
)

Converts generated QA JSON into rubric records used by the evaluation metric.

Rubric validator metric

pointwise_metric = PointwiseMetric(
    metric="gecko_metric",
    metric_prompt_template=RUBRIC_VALIDATOR_PROMPT,
    custom_output_config=CustomOutputConfig(
        return_raw_output=True,
        parsing_fn=parse_rubric_results,
    ),
)

Uses a custom output parser to extract validator answers from raw metric output.

Run EvalTask

eval_task = EvalTask(
    dataset=dataset_with_rubrics,
    metrics=[rubric_based_gecko],
)
eval_result = eval_task.evaluate(response_column_name="video")

Runs the rubric-based Gecko metric over video responses in the dataset.

Models & APIs used

  • APIs / services: Vertex AI, Cloud Storage
  • SDKs / libraries: google-cloud-aiplatform, vertexai, pandas, numpy

When to use this

Use this pattern when video generations need prompt-specific, question-based quality scoring rather than a fixed rubric.

Gotchas & caveats

  • Requires an existing Google Cloud project with the Vertex AI API enabled.
  • The notebook uses billable Vertex AI components.
  • Colab users must authenticate and restart the runtime after installing packages.
  • The default location is us-central1.
  • Gecko outputs need custom parsing beyond predefined rubric metrics.
  • Validator reliability should be checked, and questions may need manual additions for an application.

Best practices

  • Generate rubrics from the prompt so metrics reflect prompt-specific challenges.
  • Separate rubric generation from the validator step.
  • Use custom parsing functions for rubric generation and validation outputs.
  • Inspect generated questions and validator reliability before relying on the score.
  • Include matching and counterexample prompts to demonstrate score differences.

Graph View

Backlinks