Generative AI Google Brain

gemini-evaluation-create-agent-and-run-evaluation

3 min read

Create & Deploy Agent and Run Gen AI Agent Evaluation

Source notebook

Repo path: gemini/evaluation/create_agent_and_run_evaluation.ipynb · Open on GitHub · intermediate

Creates, deploys, runs, and evaluates an ecommerce ADK agent on Vertex AI Agent Engine.

Summary

This notebook teaches how to define a simple ecommerce Agent Development Kit agent with tool functions, deploy it to Agent Engine, and run inference on a prompt dataset. It then demonstrates Gen AI Agent Evaluation using both a persisted Evaluation Management Service run and an optional local evaluation flow.

Key code patterns

Configure Vertex AI and GenAI client

vertexai.init(project=PROJECT_ID, location=LOCATION)
client = Client(
    project=PROJECT_ID,
    location=LOCATION,
    http_options=genai_types.HttpOptions(api_version="v1beta1"),
)

Initializes Vertex AI and a Gen AI client for Agent Engine and evaluation APIs.

Define ADK agent with tools

ecommerce_agent = Agent(
    model="gemini-2.5-flash",
    name="ecommerce_agent",
    instruction="You are an ecommerce expert",
    tools=[search_products, get_product_details, add_to_cart],
)

Combines a Gemini model, developer instruction, and Python tool functions into an ADK agent.

Deploy to Agent Engine

app = vertexai.agent_engines.AdkApp(agent=ecommerce_agent)
agent_engine = client.agent_engines.create(
    agent=app,
    config={"staging_bucket": STAGING_BUCKET,
            "requirements": ["google-cloud-aiplatform[adk,agent_engines]"]},
)

Packages the ADK app and deploys it with a staging bucket and runtime requirements.

Run inference for evaluation data

agent_dataset_with_inference = client.evals.run_inference(
    agent=agent_engine_resource_name,
    src=ecommerce_dataset,
)

Adds real agent responses and intermediate events to the evaluation dataset.

Create persisted evaluation run

evaluation_run = client.evals.create_evaluation_run(
    dataset=agent_dataset_with_inference,
    agent_info=ecommerce_agent_info,
    metrics=[types.RubricMetric.FINAL_RESPONSE_QUALITY,
             types.RubricMetric.TOOL_USE_QUALITY],
    dest=GCS_DEST,
)

Runs managed Gen AI Agent Evaluation and persists results for later access.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI, Agent Engine, Cloud Storage, Gen AI Eval SDK
  • SDKs / libraries: google-cloud-aiplatform, google-adk, google-genai, vertexai, pandas, google-cloud-storage

When to use this

Use this pattern when you need to evaluate a deployed ADK agent with rubric metrics and inspect response quality, tool use, hallucination, and safety.

Gotchas & caveats

  • Vertex AI API must be enabled for the project.
  • PROJECT_ID, LOCATION, and GCS_DEST must be provided or set through environment variables.
  • A Cloud Storage bucket is required for staging and persisted evaluation output.
  • Agent Engine deployment may take up to 10 minutes.
  • The notebook uses v1beta1 HTTP options for the Gen AI client.
  • Session inputs are required for traces.

Best practices

  • Create a small agent-specific evaluation dataset before running evaluation.
  • Run inference first so the dataset contains response and intermediate_events columns.
  • Use AgentInfo.load_from_agent with the agent definition and deployed resource name.
  • Persist managed evaluation results to Cloud Storage when results need to be retrieved later.
  • Poll evaluation runs until they reach SUCCEEDED, FAILED, or CANCELLED before displaying final results.

Graph View

Backlinks