Generative AI Google Brain

gemini-evaluation-create-genai-agent-evaluation

3 min read

Create a Gen AI Agent Evaluation for a Deployed Agent

Source notebook

Repo path: gemini/evaluation/create_genai_agent_evaluation.ipynb · Open on GitHub · intermediate

Runs inference and creates a persisted Gen AI Agent Evaluation for a deployed Vertex AI agent.

Summary

The notebook demonstrates how to evaluate a deployed agent with the Gen AI Eval SDK. It configures Vertex AI, Cloud Storage output, an agent resource, agent tool metadata, and a prompt dataset with session inputs. It then runs agent inference, creates an evaluation run with rubric metrics, polls until completion, and displays persisted results including summary metrics, agent info, detailed results, and traces.

Key code patterns

Initialize clients

vertexai.init(project=PROJECT_ID, location=LOCATION)
client = Client(
    project=PROJECT_ID,
    location=LOCATION,
    http_options=genai_types.HttpOptions(api_version="v1beta1"),
)

Sets project, region, and API version before using evals APIs.

Create evaluation bucket

storage_client = storage.Client(project=project_id)
if storage_client.lookup_bucket(bucket_name) is None:
    storage_client.create_bucket(bucket_name)
return f"gs://{path_no_prefix}"

Ensures a Cloud Storage destination exists for persisted datasets and results.

Describe agent tools

search_products = genai_types.FunctionDeclaration(
    name="search_products",
    description="Searches for products based on a query.",
    parameters={"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]},
)

Captures tool declarations needed for agent evaluation context.

Build agent dataset

session_inputs = types.evals.SessionInput(user_id="user_123", state={})
agent_dataset = pd.DataFrame({
    "prompt": agent_prompts,
    "session_inputs": [session_inputs] * len(agent_prompts),
})

Provides prompts and session inputs required to produce trace-aware evaluation data.

Run inference and evaluation

agent_dataset_with_inference = client.evals.run_inference(agent=AGENT, src=agent_dataset)
evaluation_run = client.evals.create_evaluation_run(
    dataset=agent_dataset_with_inference,
    agent=AGENT,
    agent_info=agent_info,
    metrics=[types.RubricMetric.FINAL_RESPONSE_QUALITY, types.RubricMetric.TOOL_USE_QUALITY],
    dest=GCS_DEST,
)

Adds responses and traces, then creates a persisted evaluation run.

Models & APIs used

  • APIs / services: Vertex AI, Cloud Storage, Evaluation Management Service, Agent Engine
  • SDKs / libraries: google-cloud-aiplatform[evaluation], vertexai, google-cloud-storage, google-genai, pandas

When to use this

Use this pattern when you already have a deployed agent and need persisted rubric-based evaluation with responses and traces.

Gotchas & caveats

  • Requires an existing Google Cloud project with the Vertex AI API enabled.
  • PROJECT_ID, LOCATION, GCS_DEST, and AGENT must be manually configured.
  • The notebook installs google-cloud-aiplatform[evaluation] and says to restart runtime after installation.
  • AgentInfo must be manually updated with agent-specific instructions and tools.
  • session_inputs are required for traces.
  • The client is configured with api_version=“v1beta1”.

Best practices

  • Run inference first so the dataset includes intermediate_events and response columns before evaluation.
  • Persist evaluation datasets and results to a Cloud Storage destination.
  • Provide AgentInfo with agent instruction and tool definitions for agent evaluation.
  • Poll the evaluation run until SUCCEEDED, FAILED, or CANCELLED before retrieving full results.
  • Retrieve with include_evaluation_items=True to inspect detailed evaluation items.

Graph View

Backlinks