Generative AI Google Brain

gemini-evaluation-evaltask-approach-evaluate-rag-gen-ai-evaluation-service-sdk

3 min read

Evaluate Generated Answers from Retrieval-Augmented Generation (RAG) for Question Answering with Gen AI Evaluation Service SDK

Source notebook

Repo path: gemini/evaluation/evaltask_approach/evaluate_rag_gen_ai_evaluation_service_sdk.ipynb · Open on GitHub · intermediate

Evaluates BYO RAG QA answers with Vertex AI Gen AI Evaluation, custom metrics, and result visualizations.

Summary

This notebook teaches how to evaluate retrieval-augmented generation answers for question answering using the Vertex AI Python SDK for Gen AI Evaluation. It builds reference-free and referenced evaluation datasets, defines custom pointwise metrics, runs EvalTask evaluations, and displays summary tables, plots, and per-instance explanations.

Key code patterns

Initialize Vertex AI

PROJECT_ID = "[your-project-id]"
LOCATION = "us-central1"
EXPERIMENT = "rag-eval-01"
 
import vertexai
vertexai.init(project=PROJECT_ID, location=LOCATION)

Sets the Google Cloud project and region before running evaluation tasks.

Build BYO response dataset

eval_dataset_rag_a = pd.DataFrame({
    "prompt": ["Answer the question: " + q + " Context: " + c for q, c in zip(questions, retrieved_contexts)],
    "response": generated_answers_by_rag_a,
})

Shows the required prompt and response fields for reference-free RAG answer evaluation.

Create custom pointwise metrics

relevance = PointwiseMetric(
    metric="relevance",
    metric_prompt_template=relevance_prompt_template,
)
helpfulness = PointwiseMetric(
    metric="helpfulness",
    metric_prompt_template=helpfulness_prompt_template,
)

Demonstrates rubric-based custom model-evaluated metrics with prompt templates.

Run reference-free evaluation

rag_eval_task_rag_a = EvalTask(
    dataset=eval_dataset_rag_a,
    metrics=["question_answering_quality", relevance, helpfulness, "groundedness", "safety", "instruction_following"],
    experiment=EXPERIMENT,
)
result_rag_a = rag_eval_task_rag_a.evaluate()

Combines predefined and custom metrics to evaluate generated answers without golden answers.

Run referenced evaluation

referenced_answer_eval_task_rag_a = EvalTask(
    dataset=referenced_eval_dataset_rag_a,
    metrics=[question_answering_correctness, "rouge", "bleu", "exact_match"],
    experiment=EXPERIMENT,
)
referenced_result_rag_a = referenced_answer_eval_task_rag_a.evaluate()

Evaluates responses against golden answers using custom correctness plus lexical metrics.

Models & APIs used

  • APIs / services: Vertex AI, Gen AI Evaluation Service
  • SDKs / libraries: google-cloud-aiplatform[evaluation], vertexai, pandas

When to use this

Use this pattern when you already have RAG prompts, retrieved context, generated answers, and optionally golden answers to evaluate QA quality programmatically.

Gotchas & caveats

  • Requires installing google-cloud-aiplatform[evaluation] and restarting the notebook runtime.
  • Colab authentication is required when running in Google Colab.
  • PROJECT_ID must be set before vertexai.init or the notebook raises ValueError.
  • The notebook uses LOCATION = “us-central1”.
  • Online evaluation quotas may affect performance and user experience.
  • Referenced evaluation requires a reference column with golden answers.

Best practices

  • Use at least one evaluation example, with around 100 examples recommended for high-quality aggregated metrics and statistical significance.
  • Use reference-free evaluation when assessing generated answers against retrieved context without golden answers.
  • Use referenced evaluation when golden answers are available for comparison.
  • Inspect predefined metric templates before selecting metrics.
  • Use detailed explanations to understand why scores were assigned for individual instances.
  • Compare multiple RAG answer sets with the same datasets and metrics.

Graph View

Backlinks