Generative AI Google Brain

gemini-evaluation-evaltask-approach-multimodal-text-quality-rubric-evaluation

3 min read

Rubric evaluation - Multimodal and Custom metric for text quality

Source notebook

Repo path: gemini/evaluation/evaltask_approach/multimodal_text_quality_rubric_evaluation.ipynb · Open on GitHub · intermediate

Evaluates multimodal car-damage responses and text summaries with Vertex AI rubric-based metrics

Summary

This notebook teaches rubric-based evaluation with the Vertex AI Gen AI Evaluation service. It first runs a predefined pairwise multimodal understanding metric over pre-generated car-damage assessment responses and baselines. It then builds a custom text-quality rubric metric, generates rubrics, optionally inspects them, and runs pointwise evaluation with gemini-2.5-flash.

Key code patterns

Initialize Vertex AI and bucket

PROJECT_ID = "[your-project-id]"
LOCATION = "us-central1"
BUCKET_URI = f"gs://{BUCKET_NAME}"
!gsutil mb -l {LOCATION} {BUCKET_URI}
vertexai.init(project=PROJECT_ID, location=LOCATION)

Sets the project, region, and Cloud Storage URI used by Vertex AI evaluation jobs.

Predefined pairwise multimodal evaluation

eval_task = EvalTask(
    dataset=eval_dataset,
    metrics=[PredefinedRubricMetrics.Pairwise.MULTIMODAL_UNDERSTANDING],
    output_uri_prefix=BUCKET_URI,
)
 
eval_result = eval_task.evaluate()

Uses an off-the-shelf rubric metric to compare candidate and baseline multimodal responses.

Custom rubric-based text metric

rbm = RubricBasedMetric(
    generation_config=RubricGenerationConfig(
        prompt_template=rubric_gen_prompt,
        parsing_fn=utils.parse_rubrics,
    ),
    critique_metric=PointwiseMetric(
        metric="custom_rubric_based_text_quality",
        metric_prompt_template=rubric_critique_prompt,
        custom_output_config=CustomOutputConfig(
            return_raw_output=True,
            parsing_fn=utils.parse_pointwise_rubric_result,
        ),
    ),
)

Separates rubric generation from critique scoring and preserves raw autorater output for debugging.

Generate rubrics then evaluate

data_with_rubrics = rbm.generate_rubrics(eval_dataset)
 
eval_task = EvalTask(
    dataset=data_with_rubrics,
    metrics=[rbm],
)
 
eval_result = eval_task.evaluate(model="gemini-2.5-flash")

Reuses generated rubrics so evaluation moves directly to pointwise critique with Gemini responses.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI, Vertex AI Gen AI Evaluation service, Cloud Storage
  • SDKs / libraries: google-cloud-aiplatform[evaluation], vertexai, pandas

When to use this

Use this pattern when you need automated LLM-as-rater evaluation for multimodal comparisons or custom text-quality scoring on Vertex AI.

Gotchas & caveats

  • A Google Cloud project is required and the Vertex AI API must be enabled.
  • Colab authentication is only run when google.colab is present.
  • The notebook creates and uses a Cloud Storage bucket for evaluation output.
  • Vertex AI Gen AI Evaluation does not yet support direct multimodal inference within EvalTask, so the multimodal section uses pre-generated responses.
  • Custom rubric prompts must match the parsing functions used for rubrics and pointwise verdicts.

Best practices

  • Inspect, edit, or add generated rubrics before final evaluation.
  • Use predefined rubric metrics when they fit the multimodal task.
  • Use custom prompt templates when text-quality criteria must be controlled.
  • Return raw autorater output to debug reasoning and verdicts.
  • Write rubrics as granular binary yes/no constraints and avoid hallucinated or repeated criteria.

Graph View

Backlinks