Generative AI Google Brain

gemini-evaluation-evaltask-approach-rubric-based-eval

2 min read

Rubric-based instruction following evaluation using Gen AI Evaluation Service

Source notebook

Repo path: gemini/evaluation/evaltask_approach/rubric_based_eval.ipynb · Open on GitHub · intermediate

Evaluates Gemini instruction following with rubric-based metrics in Vertex AI.

Summary

This notebook teaches rubric-based evaluation for instruction following with Vertex AI Gen AI Evaluation Service. It builds a pandas prompt dataset, generates pointwise instruction-following rubrics, optionally revises them in Colab, and evaluates Gemini responses with EvalTask. It also shows a proprietary rubric_based_instruction_following metric and displays per-rubric scores and raw autorater outputs.

Key code patterns

Initialize Vertex AI

PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")
LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "us-central1")
vertexai.init(project=PROJECT_ID, location=LOCATION)

Sets project and region before using Vertex AI evaluation.

Generate rubrics

eval_dataset = pd.DataFrame({"prompt": prompt})
metric = PredefinedRubricMetrics.Pointwise.INSTRUCTION_FOLLOWING
data_with_rubrics = metric.generate_rubrics(eval_dataset)

Creates instruction-following rubrics from prompts before evaluation.

Evaluate with rubrics

eval_task = EvalTask(
    dataset=data_with_rubrics,
    metrics=[metric],
)
eval_result = eval_task.evaluate(model="gemini-2.5-flash")

Runs pointwise rubric critiquing against a Gemini model.

Use proprietary metric

eval_task = EvalTask(
    dataset=eval_dataset,
    metrics=["rubric_based_instruction_following"],
)
eval_result = eval_task.evaluate(model="gemini-2.5-pro")

Lets EvalTask generate and apply instruction-following rubrics in one step.

Models & APIs used

  • Models: gemini-2.5-flash, gemini-2.5-pro
  • APIs / services: Vertex AI
  • SDKs / libraries: google-cloud-aiplatform[evaluation], vertexai, pandas, google.colab

When to use this

Use this pattern to score whether Gemini outputs follow prompt instructions using generated yes/no rubrics.

Gotchas & caveats

  • Vertex AI is billable and Vertex AI API must be enabled.
  • PROJECT_ID must be set directly or via GOOGLE_CLOUD_PROJECT.
  • Default region is us-central1 unless GOOGLE_CLOUD_REGION is set.
  • Colab authentication and InteractiveSheet revision only apply when running in Google Colab.

Best practices

  • Use predefined rubric metrics for supported use cases such as Instruction Following, Multimodal Understanding, and Text Quality.
  • Generate rubrics first when you want to review and revise them before scoring responses.
  • Display evaluation results to inspect rubrics, score, rubric_verdict_pairs, and raw_outputs.
  • Use environment variables for project and region defaults in notebook setup.

Graph View

Backlinks