Generative AI Google Brain

gemini-evaluation-evaltask-approach-evaluating-langgraph-agent

3 min read

Evaluating Agents - Evaluate a LangGraph agent with Vertex AI Gen AI Evaluation Service

Source notebook

Repo path: gemini/evaluation/evaltask_approach/evaluating_langgraph_agent.ipynb · Open on GitHub · advanced

Evaluates a LangGraph Gemini agent with Vertex AI Gen AI Evaluation metrics and BYOD data.

Summary

This notebook builds a local LangGraph customer-support agent using ChatVertexAI with Gemini and two product tools. It prepares an evaluation dataset with prompts and reference trajectories, then runs Vertex AI Gen AI Evaluation for single tool use, trajectory matching, response quality, and a custom pointwise metric. It also demonstrates a bring-your-own-dataset flow with predicted trajectories and responses supplied directly.

Key code patterns

Initialize Vertex AI experiment

PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")
LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "us-central1")
EXPERIMENT_NAME = "evaluate-langgraph-agent"
vertexai.init(project=PROJECT_ID, location=LOCATION, experiment=EXPERIMENT_NAME)

Sets the project, region, and experiment used by evaluation runs.

Define LangChain tools

@tool
def get_product_details(product_name: str):
    return details.get(product_name, "Product details not found.")
 
@tool
def get_product_price(product_name: str):
    return details.get(product_name, "Product price not found.")

Creates tool functions the LangGraph agent can call for product support tasks.

Build queryable LangGraph agent

model = ChatVertexAI(model=llm)
model_with_tools = model.bind_tools([get_product_details, get_product_price])
builder = MessageGraph()
builder.add_node("tools", model_with_tools)
builder.add_conditional_edges("tools", router)
app = builder.compile()

Wraps a Gemini chat model and tools in a LangGraph flow for evaluation.

Run EvalTask with runnable

eval_task = EvalTask(
    dataset=eval_sample_dataset,
    metrics=trajectory_metrics,
    experiment=EXPERIMENT_NAME,
    output_uri_prefix=BUCKET_URI + "/multiple-metric-eval",
)
result = eval_task.evaluate(runnable=agent_parsed_outcome, experiment_run_name=EXPERIMENT_RUN)

Evaluates the live agent function against dataset rows and selected metrics.

Create custom pointwise metric

template = PointwiseMetricPromptTemplate(
    criteria=criteria,
    rating_rubric=pointwise_rating_rubric,
    input_variables=["prompt", "predicted_trajectory"],
)
metric = PointwiseMetric(metric="response_follows_trajectory", metric_prompt_template=template)

Defines a model-based metric that checks whether responses follow tool trajectories.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI Gen AI Evaluation, Vertex AI
  • SDKs / libraries: google-cloud-aiplatform[evaluation], vertexai, langchain_google_vertexai, langgraph, langchain

When to use this

Use this pattern to evaluate a LangGraph agent’s tool choices, trajectories, and responses with Vertex AI Gen AI Evaluation.

Gotchas & caveats

  • Vertex AI API must be enabled for the Google Cloud project.
  • The notebook installs packages and requires a runtime restart before continuing.
  • Colab requires explicit user authentication with google.colab.auth.authenticate_user().
  • PROJECT_ID falls back to GOOGLE_CLOUD_PROJECT and LOCATION defaults to us-central1.
  • EvalTask examples use BUCKET_URI for output_uri_prefix, so a bucket URI must be available in the notebook environment.
  • BYOD evaluation requires predicted_trajectory and reference_trajectory serialized with json.dumps.

Best practices

  • Use an experiment name when initializing Vertex AI to organize evaluation runs.
  • Include prompts and reference trajectories in the evaluation dataset for agent trajectory metrics.
  • Evaluate single tool use before broader trajectory and response evaluations.
  • Use multiple trajectory metrics to compare exact order, in-order match, any-order match, precision, and recall.
  • Use custom pointwise metrics when response quality depends on the agent’s tool trajectory.
  • Delete the experiment and backing TensorBoard runs during cleanup when delete_experiment is true.

Graph View

Backlinks