agents-adk-new-hire-onboarding-tests-eval-evalsets-readme

Repository doc · agents/adk/new-hire-onboarding/tests/eval/evalsets/README.md · Open on GitHub · Part of Agents & ADK Notebooks

Evaluation Sets

This directory contains evaluation sets for testing agent behavior using adk eval.

Running Evaluations

# Run default evalset
agents-cli eval run
 
# Run specific evalset
agents-cli eval run --evalset tests/eval/evalsets/custom.evalset.json
 
# Run all evalsets
agents-cli eval run --all

Evalset Format

Each .evalset.json follows the ADK evaluation format:

{
  "eval_set_id": "unique_id",
  "name": "Human-readable name",
  "description": "What this evalset tests",
  "eval_cases": [
    {
      "eval_id": "case_id",
      "conversation": [
        {
          "user_content": {
            "parts": [{"text": "User message"}]
          },
          "intermediate_data": {
            "tool_uses": [
              {"name": "tool_name", "args": {"param": "value"}}
            ]
          }
        }
      ],
      "session_input": {
        "app_name": "app_name",
        "user_id": "test_user",
        "state": {}
      }
    }
  ]
}

Key Fields

• eval_cases: Array of test scenarios
• conversation: Sequence of user messages
• intermediate_data.tool_uses: Expected tool calls (for trajectory matching)
• session_input: Initial session state

Evaluation Metrics

ADK eval measures:

• tool_trajectory_avg_score: Are the correct tools called in the right order?
• response_match_score: How similar is the response to expected output?

Creating Custom Evalsets

1. Copy basic.evalset.json as a template
2. Add cases based on your DESIGN_SPEC.md scenarios
3. Include expected tool calls for capability tests
4. Run agents-cli eval run --evalset your_evalset.json

Tips

• Start with 3-5 representative cases
• Include both happy path and edge cases
• Test each core capability from DESIGN_SPEC.md
• Add cases when you find bugs in production

See ADK documentation for advanced evaluation options.

---

Repository Docs · Home