Generative AI Google Brain

gemini-evaluation-evaltask-approach-evaluate-agent-final-answer-with-custom-parsing

2 min read

Evaluate agent final answer with custom parsing

Source notebook

Repo path: gemini/evaluation/evaltask_approach/evaluate_agent_final_answer_with_custom_parsing.ipynb · Open on GitHub · intermediate

Evaluates agent final answers using Vertex AI EvalTask with a custom JSON parser for autorater output.

Summary

The notebook teaches how to evaluate whether an agent final answer matches a human reference answer using the Vertex Gen AI Evaluation SDK. It defines a pointwise metric with an autorater prompt that requests JSON, parses that structured output with a custom parsing function, runs EvalTask on a small dataset, and computes an overall validity score from the parsed results.

Key code patterns

Initialize Vertex AI

PROJECT_ID = "your-project-id"
LOCATION = "us-central1"
 
import vertexai
vertexai.init(project=PROJECT_ID, location=LOCATION)

Sets the Google Cloud project and region before using Vertex AI evaluation.

Parse autorater JSON

def parse_response_to_json(responses: list[str]) -> dict[str, Any]:
    response = re.sub(r"(.*```json|```.*)", "", responses[0].strip())
    result = None
    try:
        result = json.loads(response)
    except Exception as e:
        print(f"Failed to parse JSON response: {e!s}")
    return {"result": result}

Converts the autorater’s structured text output into a dictionary appended to evaluation results.

Define custom metric

agent_final_answer_metric = PointwiseMetric(
    metric="agent_final_answer",
    metric_prompt_template=AGENT_FINAL_ANSWER_PROMPT,
    custom_output_config=CustomOutputConfig(
        return_raw_output=True,
        parsing_fn=parse_response_to_json,
    ),
)

Combines a metric prompt with CustomOutputConfig so parsed JSON becomes part of the metrics table.

Run evaluation

eval_task = EvalTask(
    dataset=eval_dataset,
    metrics=[agent_final_answer_metric],
    autorater_config=AutoraterConfig(sampling_count=1),
)
eval_result = eval_task.evaluate()
compute_metric_score(eval_result.metrics_table)

Runs pointwise evaluation and computes the share of valid agent responses.

Models & APIs used

  • APIs / services: Vertex AI
  • SDKs / libraries: google-cloud-aiplatform, vertexai, pandas

When to use this

Use this pattern when evaluating agent answers that need structured autorater judgments beyond standard metric templates.

Gotchas & caveats

  • Vertex AI API must be enabled for the Google Cloud project.
  • The tutorial uses billable Vertex AI components.
  • Google Colab requires authentication and a runtime restart after installing packages.
  • The parser assumes the autorater response is valid JSON or fenced JSON text.
  • PROJECT_ID must be replaced with an existing Google Cloud project.

Best practices

  • Use a human reference response as the golden answer for judging agent output validity.
  • Request structured JSON from the autorater when downstream parsing is needed.
  • Use CustomOutputConfig with a parsing function to append parsed output to evaluation results.
  • Return raw autorater output alongside parsed results for inspection.
  • Separate helper functions, prompt template, metric definition, dataset preparation, and evaluation execution.

Graph View

Backlinks