Generative AI Google Brain

gemini-evaluation-evaluating-adk-agent

3 min read

Evaluate your ADK agent using Vertex AI Gen AI Evaluation service

Source notebook

Repo path: gemini/evaluation/evaluating_adk_agent.ipynb · Open on GitHub · intermediate

Evaluates an ADK product research agent with Vertex AI Gen AI Evaluation metrics.

Summary

The notebook builds a local ADK agent using Gemini and two custom product tools, then prepares an evaluation dataset with prompts and reference tool trajectories. It runs Vertex AI Gen AI Evaluation for single tool use, trajectory matching, response quality, and a custom pointwise metric that checks whether responses follow tool choices. It also demonstrates a bring-your-own-dataset flow with precomputed responses and predicted trajectories.

Key code patterns

Initialize Vertex AI evaluation

os.environ["GOOGLE_GENAI_USE_VERTEXAI"] = "True"
vertexai.init(
    project=PROJECT_ID,
    location=LOCATION,
    experiment=EXPERIMENT_NAME,
)

Sets project, region, Vertex AI routing, and experiment tracking before evaluations.

Build an ADK agent with tools

product_research_agent = Agent(
    name="ProductResearchAgent",
    model=model,
    instruction="Use price tool for price requests, otherwise details tool.",
    tools=[get_product_details, get_product_price],
)

Shows how the notebook wires Gemini and Python functions into a local ADK agent.

Parse ADK events for evaluation

for event in events:
    for part in event.content.parts:
        if getattr(part, "function_call", None):
            trajectory.append({
                "tool_name": part.function_call.name,
                "tool_input": dict(part.function_call.args),
            })

Extracts predicted tool calls into the trajectory format expected by evaluation metrics.

Run an EvalTask

eval_task = EvalTask(
    dataset=eval_sample_dataset,
    metrics=trajectory_metrics,
    experiment=EXPERIMENT_NAME,
    output_uri_prefix=BUCKET_URI + "/multiple-metric-eval",
)
result = eval_task.evaluate(runnable=agent_parsed_outcome_sync)

Evaluates a runnable agent against a dataframe dataset and stores results in Cloud Storage.

Define a custom pointwise metric

template = PointwiseMetricPromptTemplate(
    criteria=criteria,
    rating_rubric=pointwise_rating_rubric,
    input_variables=["prompt", "predicted_trajectory"],
)
metric = PointwiseMetric(
    metric="response_follows_trajectory",
    metric_prompt_template=template,
)

Creates a custom model-based metric for checking whether responses follow tool trajectories.

Models & APIs used

  • Models: gemini-2.0-flash
  • APIs / services: Vertex AI, Vertex AI Gen AI Evaluation service, Cloud Storage
  • SDKs / libraries: google-adk, google-cloud-aiplatform, google-genai, vertexai, pandas, plotly

When to use this

Use this pattern to evaluate an agent’s tool selection, trajectory, and response quality with Vertex AI Gen AI Evaluation.

Gotchas & caveats

  • A Google Cloud project with the Vertex AI API enabled is required.
  • The notebook creates a Cloud Storage bucket with gsutil and uses it as output_uri_prefix.
  • Colab authentication is only run when google.colab is detected.
  • LOCATION defaults to us-central1 from GOOGLE_CLOUD_REGION if not provided.
  • The synchronous evaluation wrapper uses asyncio.run, which can matter in active event loop environments.
  • BYOD evaluation expects predicted_trajectory, reference_trajectory, and response serialized with json.dumps.

Best practices

  • Use a small reference dataset with prompts and expected tool trajectories.
  • Evaluate single tool selection before broader trajectory metrics.
  • Track evaluation runs with unique experiment run names.
  • Separate trajectory metrics from response metrics such as safety and coherence.
  • Use custom pointwise criteria when response quality depends on tool choices.
  • Clean up the Vertex AI experiment when finished.

Graph View

Backlinks