Generative AI Google Brain

gemini-evaluation-evaltask-approach-enhancing-quality-and-explainability-with-eval

3 min read

Enhancing quality and explainability with Vertex AI Evaluation

Source notebook

Repo path: gemini/evaluation/evaltask_approach/enhancing_quality_and_explainability_with_eval.ipynb · Open on GitHub · intermediate

Ranks Gemini answers with Vertex AI pairwise and pointwise evaluation explanations.

Summary

This notebook shows how to generate multiple Gemini responses through the Google GenAI SDK on Vertex AI, then choose the best answer using Vertex AI Gen AI Evaluation. It builds pairwise AutoSXS comparisons for question-answering quality, runs pointwise question-answering quality and groundedness metrics on the selected answer, and returns human-readable explanations. The workflow also tracks evaluation runs under a Vertex AI experiment and deletes the experiment during cleanup.

Key code patterns

Generate multiple Gemini candidates

client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)
responses = [
    candidate.content.parts[0].text
    for candidate in client.models.generate_content(
        model="gemini-2.5-flash",
        contents=prompt_qa,
        config=types.GenerateContentConfig(
            temperature=0.4, max_output_tokens=512, candidate_count=3
        ),
    ).candidates
]

Creates several slightly decorrelated answers in one Gemini call for later ranking.

Pairwise evaluation dataset

eval_dataset = pd.DataFrame({
    "instruction": [instructions],
    "context": [context],
    "response": [candidate],
    "baseline_model_response": [baseline],
})
eval_task = EvalTask(
    dataset=eval_dataset,
    metrics=[MetricPromptTemplateExamples.Pairwise.QUESTION_ANSWERING_QUALITY],
    experiment=experiment_name,
)

Formats baseline and candidate responses for AutoSXS-style question-answering quality comparison.

Retrieve pairwise choice and explanation

results = eval_task.evaluate(
    prompt_template="{instruction} \n {context}",
    experiment_run_name="gemini-qa-pairwise-" + str(uuid.uuid4()),
)
result = results.metrics_table[[
    "pairwise_question_answering_quality/pairwise_choice",
    "pairwise_question_answering_quality/explanation",
]].to_dict("records")[0]

Extracts both the winning side and the human-readable rationale from evaluation results.

Pointwise quality and groundedness

eval_task = EvalTask(
    dataset=eval_dataset,
    metrics=[
        MetricPromptTemplateExamples.Pointwise.QUESTION_ANSWERING_QUALITY,
        MetricPromptTemplateExamples.Pointwise.GROUNDEDNESS,
    ],
    experiment=experiment_name,
)
results = eval_task.evaluate(
    prompt_template="{instruction} \n {context}",
    experiment_run_name="gemini-qa-pointwise-" + str(uuid.uuid4()),
)

Scores selected answers individually and returns explainable QA quality and groundedness metrics.

Rank then explain best response

cmp_f = partial(pairwise_greater, instruction, context, PROJECT_ID, LOCATION, experiment_name)
cmp_greater = partial(greater, cmp_f)
pairwise_best_response = max(responses, key=functools.cmp_to_key(cmp_greater))
pointwise_metric = pointwise_eval(instruction, context, [pairwise_best_response])

Combines pairwise selection with pointwise evaluation for a best-answer-plus-metrics workflow.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI, Gemini API in Vertex AI, Vertex Gen AI Evaluation Service
  • SDKs / libraries: google-genai, google-cloud-aiplatform, vertexai, pandas, nest_asyncio, bigframes

When to use this

Use this pattern when you need to generate several LLM answers and return the best one with evaluation-backed explanations.

Gotchas & caveats

  • Requires Google Cloud project ID and Vertex AI location values before running.
  • Notebook sets LOCATION to us-central1.
  • Colab requires auth.authenticate_user(); Vertex AI Workbench does not require that step.
  • Notebook recommends restarting the runtime after installing upgraded packages.
  • Uses billable Vertex AI components.
  • Evaluation runs are stored under the qa-quality experiment unless cleaned up.
  • The demonstrated workflow is text-only, though the overview says it can extend to other modalities when evaluation exists.

Best practices

  • Generate multiple candidate responses before ranking for quality.
  • Use pairwise evaluation to select the best response among candidates.
  • Use pointwise evaluation to report quality and groundedness explanations for the selected response.
  • Use a prompt template that includes both instruction and context during evaluation.
  • Use unique experiment_run_name values with uuid.uuid4().
  • Clean up the Vertex AI experiment when finished.

Graph View

Backlinks