Generative AI Google Brain

gemini-evaluation-evaltask-approach-evaluate-images-with-gecko

3 min read

Evaluate images with Gecko

Source notebook

Repo path: gemini/evaluation/evaltask_approach/evaluate_images_with_gecko.ipynb · Open on GitHub · intermediate

Evaluates prompt-image alignment with Gecko-style rubric generation and VQA validation in Vertex AI.

Summary

This notebook teaches how to use Vertex AI evaluation with a Gecko-style two-stage workflow for image evaluation. It generates prompt-specific question-answer rubrics, validates generated images against those questions with a Gemini model, then computes final and per-question scores. The example builds a prompt/image dataset from Cloud Storage image URIs and runs EvalTask with a custom RubricBasedMetric.

Key code patterns

Initialize Vertex AI

PROJECT_ID = "your-project-id"
LOCATION = "us-central1"
 
import vertexai
vertexai.init(project=PROJECT_ID, location=LOCATION)

Sets the Google Cloud project and region before using Vertex AI evaluation.

Parse generated rubrics

def parse_json_to_qa_records(json_response):
    json_response = re.sub(r"(.*```json|```.*)", "", json_response.strip())
    data = json.loads(json_response)
    qa_records = []
    for qa in data["qas"]:
        qa_records.append(QARecord(
            question=qa["question"],
            question_type=qa["question_type"],
            gt_answer=qa["answer"],
            answer_choices=qa["choices"],
            justification=qa["justification"],
        ))

Custom parsing converts rubric-generation JSON into QARecord objects used by the metric.

Build Gecko metric

rubric_generation_config = RubricGenerationConfig(
    prompt_template=RUBRIC_GENERATION_PROMPT,
    parsing_fn=parse_json_to_qa_records,
)
pointwise_metric = PointwiseMetric(
    metric="gecko_metric",
    metric_prompt_template=RUBRIC_VALIDATOR_PROMPT,
    custom_output_config=CustomOutputConfig(return_raw_output=True, parsing_fn=parse_rubric_results),
)
rubric_based_gecko = RubricBasedMetric(generation_config=rubric_generation_config, critique_metric=pointwise_metric)

Combines prompt-specific rubric generation with pointwise validation over image responses.

Run evaluation

eval_dataset = pd.DataFrame({"prompt": prompts, "image": images})
dataset_with_rubrics = rubric_based_gecko.generate_rubrics(eval_dataset)
eval_task = EvalTask(dataset=dataset_with_rubrics, metrics=[rubric_based_gecko])
eval_result = eval_task.evaluate(response_column_name="image")
dataset_with_final_scores = compute_scores(eval_result.metrics_table)
np.mean(dataset_with_final_scores["final_score"])

Separates rubric generation from image evaluation and aggregates final scores.

Models & APIs used

  • APIs / services: Vertex AI, Cloud Storage
  • SDKs / libraries: google-cloud-aiplatform, vertexai, pandas, numpy

When to use this

Use this pattern when evaluating whether generated images satisfy prompt-specific visual requirements with custom QA rubrics.

Gotchas & caveats

  • Vertex AI is billable and the Vertex AI API must be enabled for the project.
  • Google Colab requires installing google-cloud-aiplatform, restarting the runtime, and authenticating the user.
  • The example initializes Vertex AI in us-central1.
  • Gecko outputs need custom parsing beyond predefined rubric-based metric defaults.
  • Image inputs are JSON strings with file_data containing mime_type and a gs:// file_uri.

Best practices

  • Generate rubrics from each prompt before validation so the metric adapts to prompt-specific challenges.
  • Use CustomOutputConfig with return_raw_output=True and parsing_fn for custom validator outputs.
  • Compare matching prompts with similar counterexample prompts to show high-quality and low-quality responses.
  • Inspect generated questions and validator reliability, and manually add questions when needed.
  • Aggregate validation results into final scores and per-question QA results.

Graph View

Backlinks