Generative AI Google Brain

gemini-evaluation-multi-turn-agent-evaluation-with-user-simulation-metric-registration-auto-loss-analysis

3 min read

Gen AI Eval - Multi-turn Agent Eval, User Simulation, Metric Registration, Auto-Loss Analysis

Source notebook

Repo path: gemini/evaluation/multi_turn_agent_evaluation_with_user_simulation_metric_registration_auto_loss_analysis.ipynb · Open on GitHub · advanced

Evaluates ADK travel agents with Vertex Gen AI Eval multi-turn simulation, custom metrics, and loss analysis.

Summary

The notebook teaches how to evaluate a Google ADK travel agent locally and as a deployed Agent Engine app using Vertex Gen AI Evaluation SDK. It generates synthetic multi-turn scenarios, runs user simulation, registers custom code and LLM metrics, evaluates predefined multi-turn metrics, and groups failures with auto-loss analysis.

Key code patterns

Initialize Vertex AI eval client

client = Client(
    project=PROJECT_ID,
    location=LOCATION,
)
os.environ["GOOGLE_GENAI_USE_VERTEXAI"] = "1"
os.environ["GOOGLE_CLOUD_PROJECT"] = PROJECT_ID
os.environ["GOOGLE_CLOUD_LOCATION"] = LOCATION

Configures the notebook to use Vertex AI Agent Platform and the selected project and region.

Create ADK orchestrator agent

travel_agent = Agent(
    model="gemini-3.5-flash",
    name="travel_agent",
    instruction="You are a primary travel concierge...",
    sub_agents=[flight_agent, hotel_agent],
)

Builds a root travel concierge that delegates flight and hotel tasks to sub-agents.

Generate user simulation scenarios

agent_info = types.evals.AgentInfo.load_from_agent(agent=travel_agent)
eval_dataset = client.evals.generate_conversation_scenarios(
    agent_info=agent_info,
    config={"count": 5, "generation_instruction": "Generate scenarios where the user tries to book a flight but changes their mind about the destination."},
)

Creates synthetic multi-turn evaluation cases from the agent description and scenario instructions.

Run local multi-turn inference

eval_dataset_with_trace = client.evals.run_inference(
    agent=travel_agent,
    src=eval_dataset,
    config={"user_simulator_config": {"max_turn": 3}},
)

Uses the user simulator to collect conversation traces from the local ADK agent.

Register custom metric

efficiency_metric = types.CodeExecutionMetric(
    name="multi_turn_efficiency",
    custom_function=efficiency_metric_code,
)
efficiency_metric_path = client.evals.create_evaluation_metric(metric=efficiency_metric)

Stores a reusable custom computation metric in the evaluation metric registry.

Evaluate and analyze losses

eval_result = client.evals.evaluate(
    dataset=eval_dataset_with_trace,
    metrics=eval_metrics,
    config={"evaluation_service_qps": 5.0},
)
loss_analysis = client.evals.generate_loss_clusters(
    eval_result=eval_result,
    metric=types.RubricMetric.MULTI_TURN_TOOL_USE_QUALITY,
)

Scores traces with predefined and custom metrics, then clusters failed traces into semantic loss patterns.

Models & APIs used

When to use this

Use this pattern when validating multi-turn ADK agents with simulated users, reusable custom metrics, and failure clustering before or after Agent Engine deployment.

Gotchas & caveats

  • For local agent scraping with gemini-3 models, use the global region.
  • Agent Engine cannot be deployed in the global region; the notebook deploys to us-central1.
  • Cloud Agent Eval requires GCS buckets for Agent Engine staging and evaluation run output.
  • generate_conversation_scenarios defaults to gemini-3.1-pro-preview and may take several minutes.
  • The user simulator defaults to gemini-2.5-flash when no model_name is provided.
  • Increase max_turn when hitting turn-limit warnings.
  • Use evaluation_service_qps to reduce quota exhausted errors.
  • Single-turn metrics do not work with simulated multi-turn agent_data unless max_turn is 1.
  • Auto-loss Analysis is only supported for MULTI_TURN_TASK_SUCCESS and MULTI_TURN_TOOL_USE_QUALITY in this notebook.

Best practices

  • Load AgentInfo from the ADK agent before scenario generation.
  • Throttle evaluation calls with evaluation_service_qps.
  • Register custom metrics once and reference their metric_resource_name in evaluation runs.
  • Return a dictionary from custom LLM metric result_parsing_function.
  • Use predefined multi-turn metrics for tool use quality, trajectory quality, and task success.
  • Poll long-running evaluation runs until a terminal state before retrieving items.

Graph View

Backlinks