Generative AI Google Brain

gemini-agent-engine-evaluating-langgraph-agent-engine-customized-template

3 min read

Evaluate a LangGraph agent on Vertex AI Agent Engine (Customized template)

Source notebook

Repo path: gemini/agent-engine/evaluating_langgraph_agent_engine_customized_template.ipynb · Open on GitHub · advanced

Builds a Gemini LangGraph agent on Agent Engine and evaluates tools, trajectories, and responses.

Summary

This notebook builds a customer support LangGraph agent with Gemini and two product tools, tests it locally, and deploys it to Vertex AI Agent Engine. It prepares evaluation datasets with prompts and reference tool trajectories, then runs Vertex AI Gen AI Evaluation for single-tool use, trajectory matching, response quality, a custom pointwise metric, and a BYOD-style evaluation. It also shows experiment cleanup and remote agent deletion.

Key code patterns

Initialize Vertex AI

vertexai.init(
    project=PROJECT_ID,
    location=LOCATION,
    staging_bucket=BUCKET_URI,
    experiment=EXPERIMENT_NAME,
)

Configures project, region, staging bucket, and experiment before deployment and evaluation.

Define agent tools

@tool
def get_product_details(product_name: str):
    return details.get(product_name, "Product details not found.")
 
@tool
def get_product_price(product_name: str):
    return details.get(product_name, "Product price not found.")

Creates callable product lookup tools used by the LangGraph agent.

Route tool calls

def router(state):
    tool_calls = state[-1].tool_calls
    if tool_calls:
        function_name = tool_calls[0].get("name")
        if function_name == "get_product_price":
            return "get_product_price"
        return "get_product_details"
    return "__end__"

Directs LangGraph execution from model tool calls to the correct tool node or completion.

Build LangGraph app

model = ChatVertexAI(model=self.model)
builder = MessageGraph()
model_with_tools = model.bind_tools([get_product_details, get_product_price])
builder.add_node("tools", model_with_tools)
builder.add_conditional_edges("tools", router)
self.app = builder.compile()

Wraps a Gemini model and custom tools in a LangGraph application template.

Deploy to Agent Engine

remote_custom_agent = agent_engines.create(
    local_custom_agent,
    requirements=[
        "google-cloud-aiplatform[agent_engines,langchain]",
        "langchain_google_vertexai",
        "langgraph",
    ],
)

Packages the local custom agent and dependencies for Vertex AI Agent Engine.

Run EvalTask

eval_task = EvalTask(
    dataset=eval_sample_dataset,
    metrics=trajectory_metrics,
    experiment=EXPERIMENT_NAME,
)
result = eval_task.evaluate(runnable=agent_parsed_response)

Runs Vertex AI Gen AI Evaluation against an agent function and dataset.

Custom pointwise metric

template = PointwiseMetricPromptTemplate(
    criteria=criteria,
    rating_rubric=pointwise_rating_rubric,
    input_variables=["prompt", "predicted_trajectory"],
)
metric = PointwiseMetric(
    metric="response_follows_trajectory",
    metric_prompt_template=template,
)

Defines a model-based metric to judge whether the response follows the tool trajectory.

Models & APIs used

  • Models: gemini-2.0-flash
  • APIs / services: Vertex AI, Vertex AI Gen AI Evaluation, Vertex AI Agent Engine, Cloud Storage
  • SDKs / libraries: google-cloud-aiplatform, vertexai, langchain_google_vertexai, langgraph, crewai_tools, pandas, plotly

When to use this

Use this pattern to evaluate a deployed LangGraph Agent Engine app for tool selection, trajectory correctness, and response quality.

Gotchas & caveats

  • Vertex AI API must be enabled for the Google Cloud project.
  • A Cloud Storage staging bucket is created with gsutil and used by vertexai.init.
  • Colab requires user authentication with auth.authenticate_user().
  • Package installation requires a runtime restart, and the notebook warns that Colab may show a session crash message.
  • Agent Engine deployment is stated to take about 10 minutes.
  • The notebook pins cloudpickle==3.0.0 and requires pydantic>=2.10.
  • The default location is us-central1 unless GOOGLE_CLOUD_REGION is set.
  • BYOD section defines byod_eval_data but creates byod_eval_sample_dataset from eval_data.

Best practices

  • Evaluate agents both online and offline using subjective and objective signals.
  • Use prompts, reference responses, and reference trajectories in evaluation datasets when available.
  • Start with single tool selection, then evaluate full trajectories and generated responses.
  • Use trajectory metrics such as exact match, in-order match, any-order match, precision, and recall.
  • Parse custom agent output into response and predicted_trajectory before passing it to EvalTask.
  • Use custom PointwiseMetric criteria when text quality metrics are not sufficient for agent behavior.
  • Delete evaluation experiments and remote agents during cleanup when they are no longer needed.

Graph View

Backlinks