Generative AI Google Brain

gemini-evaluation-evaltask-approach-evaluate-groundedness-with-custom-parsing

2 min read

Evaluate groundedness with custom parsing

Source notebook

Repo path: gemini/evaluation/evaltask_approach/evaluate_groundedness_with_custom_parsing.ipynb · Open on GitHub · advanced

Evaluates Gemini response groundedness with Vertex AI EvalTask and custom JSON parsing.

Summary

This notebook teaches how to assess whether model responses are factually grounded in a supplied context. It defines an autorater prompt that labels each response sentence, parses the structured JSON output into verdicts and a score, builds a small evaluation dataset, and runs Vertex AI EvalTask with Gemini 2.5 Flash inference.

Key code patterns

Initialize Vertex AI

PROJECT_ID = "your-project-id"
LOCATION = "us-central1"
 
import vertexai
vertexai.init(project=PROJECT_ID, location=LOCATION)

Sets the Google Cloud project and region before running Vertex AI evaluation.

Parse autorater JSON

def parse_response_to_json(responses):
    response = re.sub(r"(.*```json|```.*)", "", responses[0].strip())
    sentences = json.loads(response)
    verdicts = [s["label"] for s in sentences]
    score = sum(v == "supported" for v in verdicts) / len(verdicts)
    return {"sentence": sentences, "sentence_verdict": verdicts, "model_resp_score": score}

Converts structured autorater output into dataframe columns and a groundedness score.

Define custom metric

grounded_metric = PointwiseMetric(
    metric="groundedness",
    metric_prompt_template=GROUNDING_AUTORATER_PROMPT,
    custom_output_config=CustomOutputConfig(
        return_raw_output=True,
        parsing_fn=parse_response_to_json,
    ),
)

Attaches a custom parser to a pointwise metric so raw autorater output becomes structured results.

Run EvalTask

eval_task = EvalTask(
    dataset=eval_dataset,
    metrics=[grounded_metric],
    autorater_config=AutoraterConfig(sampling_count=1),
)
 
eval_result = eval_task.evaluate(
    model="gemini-2.5-flash",
    prompt_template=RESPONSE_PROMPT_TEMPLATE,
)

Runs model inference and groundedness evaluation over the prepared query-context dataset.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI
  • SDKs / libraries: google-cloud-aiplatform, vertexai, numpy, pandas

When to use this

Use this pattern when you need sentence-level groundedness judgments and custom scoring beyond standard metric templates.

Gotchas & caveats

  • Requires an existing Google Cloud project with the Vertex AI API enabled.
  • The notebook uses billable Vertex AI components.
  • Colab users must authenticate and restart the runtime after installing packages.
  • The parser assumes the autorater returns JSON with sentence objects containing label fields.
  • Location is set to us-central1 in the notebook.

Best practices

  • Prompt the response model to answer using only the provided context.
  • Ask the autorater to be strict and avoid world knowledge unless trivial.
  • Return raw autorater output and parse it with CustomOutputConfig for detailed analysis.
  • Use structured labels for supported, unsupported, contradictory, and no_rad sentences.
  • Compute an overall groundedness score from parsed sentence verdicts.

Graph View

Backlinks