Generative AI Google Brain

gemini-evaluation-evaltask-approach-evaluate-autorater

3 min read

Evaluate your autorater with meta-evaluation

Source notebook

Repo path: gemini/evaluation/evaltask_approach/evaluate_autorater.ipynb · Open on GitHub · intermediate

Meta-evaluates Gemini autoraters on RewardBench with agreement and correlation metrics.

Summary

This notebook teaches meta-evaluation: checking how well an LLM autorater aligns with golden preference labels. The workflow installs and initializes Vertex AI, defines pairwise autorater prompts and parsers, loads RewardBench examples, and runs BasicRater and SelfConsistencyRater with gemini-2.5-flash. It aligns predictions with golden ratings and reports a confusion matrix, Cohen’s kappa, Spearman correlation, and Kendall correlation.

Key code patterns

Initialize Vertex AI

PROJECT_ID = "[your-project-id]"
if not PROJECT_ID or PROJECT_ID == "[your-project-id]":
    PROJECT_ID = str(os.environ.get("GOOGLE_CLOUD_PROJECT"))
LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "us-central1")
vertexai.init(project=PROJECT_ID, location=LOCATION)

Sets project and region before using Vertex AI-hosted Gemini calls.

Parse judge output

def simple_no_tie_result_parser(result_str):
    matches = re.findall(r"<winner>(.*?)</winner>", result_str)
    if not matches or len(matches) > 1:
        return None
    if matches[0] == "1":
        return -1
    if matches[0] == "2":
        return 1
    return None

Converts strict LLM judge tags into numeric labels for metrics.

Prepare RewardBench pairs

df = pd.read_parquet("hf://datasets/allenai/reward-bench/" + split_to_path[split])
golden_rating = random.choice([-1, 1])
if golden_rating == -1:
    response1, response2 = row["chosen"], row["rejected"]
else:
    response2, response1 = row["chosen"], row["rejected"]
id_to_example[question_id] = Example(prompt=prompt, response1=response1, response2=response2)

Builds pairwise examples while randomizing whether the chosen answer appears first or second.

Run autoraters

rater = BasicRater(autorater_model, "simple_no_tie")
id_to_basic_auto_ratings = rater.rate_batch(id_to_example)
sc_rater = SelfConsistencyRater(autorater_model, "simple_no_tie", 3)
id_to_sc_auto_ratings = sc_rater.rate_batch(id_to_example)

Compares a single-call rater with a multi-call self-consistency rater.

Score alignment

spearman, _ = spearmanr(model_outputs, golden_labels)
kendall, _ = kendalltau(model_outputs, golden_labels)
kappa = cohen_kappa_score(model_outputs, golden_labels, labels=labels, weights=weights)
conf_matrix = confusion_matrix(golden_labels, model_outputs, labels=labels)

Uses agreement and rank-correlation metrics to evaluate autorater quality.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI
  • SDKs / libraries: google-genai, vertexai, pandas, scipy, scikit-learn

When to use this

Use this pattern when validating an LLM-as-judge autorater against preference labels before relying on it for evaluation.

Gotchas & caveats

  • The notebook requires a Google Cloud project with the Vertex AI API enabled.
  • The runtime must be restarted after installing google-cloud-aiplatform and google-genai.
  • Colab runs require auth.authenticate_user().
  • The region defaults to GOOGLE_CLOUD_REGION or us-central1.
  • The RewardBench response order is randomized with random.choice, so runs are not reproducible unless seeded.
  • Parsers return None for malformed or ambiguous model output, so metrics only cover successfully parsed ratings.

Best practices

  • Compare automated evaluator outputs against golden labels before trusting the autorater.
  • Use strict output formats and parsing functions for LLM judge results.
  • Align ratings by shared sorted IDs before computing metrics.
  • Report multiple metrics: confusion matrix, Cohen’s kappa, Spearman correlation, and Kendall correlation.
  • Evaluate both a basic rater and a self-consistency rater.
  • Limit processed examples with total_count for manageable evaluation runs.

Graph View

Backlinks