Generative AI Google Brain

gemini-agent-engine-evaluating-langchain-agent-engine-prebuilt-template

3 min read

Evaluating a LangChain Agent on Vertex AI Agent Engine (Prebuilt template)

Source notebook

Repo path: gemini/agent-engine/evaluating_langchain_agent_engine_prebuilt_template.ipynb · Open on GitHub · advanced

Deploys and evaluates a LangChain Gemini agent on Vertex AI Agent Engine using Gen AI Evaluation.

Summary

This notebook builds a LangChain agent with two product lookup tools, deploys it to Vertex AI Agent Engine, and evaluates it with Vertex AI Gen AI Evaluation. It demonstrates single-tool-use checks, trajectory metrics, response metrics, a custom pointwise metric, and a BYOD evaluation dataset with predicted trajectories and responses.

Key code patterns

Initialize Vertex AI

vertexai.init(
    project=PROJECT_ID,
    location=LOCATION,
    staging_bucket=BUCKET_URI,
    experiment=EXPERIMENT_NAME,
)

Configures project, region, staging bucket, and experiment before deploying and evaluating the agent.

Create LangChain agent

local_1p_agent = LangchainAgent(
    model="gemini-2.0-flash",
    tools=[get_product_details, get_product_price],
    agent_executor_kwargs={"return_intermediate_steps": True},
)

Enables the agent to call Python tools and return intermediate steps needed for trajectory evaluation.

Deploy agent

remote_1p_agent = agent_engines.create(
    local_1p_agent,
    requirements=[
        "google-cloud-aiplatform[agent_engines,langchain]",
        "langchain_google_vertexai",
    ],
)

Packages the local LangChain agent with dependencies and deploys it to Vertex AI Agent Engine.

Evaluate runnable agent

eval_task = EvalTask(
    dataset=eval_sample_dataset,
    metrics=trajectory_metrics,
    experiment=EXPERIMENT_NAME,
)
result = eval_task.evaluate(runnable=remote_1p_agent)

Runs evaluation directly against a Queryable remote agent using a prompt and reference trajectory dataset.

Define custom pointwise metric

template = PointwiseMetricPromptTemplate(
    criteria=criteria,
    rating_rubric=pointwise_rating_rubric,
    input_variables=["prompt", "predicted_trajectory"],
)
metric = PointwiseMetric(
    metric="response_follows_trajectory",
    metric_prompt_template=template,
)

Creates a model-based metric to judge whether the final response follows the agent trajectory.

Models & APIs used

  • Models: gemini-2.0-flash
  • APIs / services: Vertex AI, Vertex AI Gen AI Evaluation, Vertex AI Agent Engine, Cloud Storage
  • SDKs / libraries: google-cloud-aiplatform, vertexai, langchain_google_vertexai, cloudpickle, pydantic, requests, pandas, plotly

When to use this

Use this pattern to evaluate whether a deployed LangChain agent selects the right tools, follows expected tool trajectories, and produces acceptable responses.

Gotchas & caveats

  • Vertex AI API must be enabled for the Google Cloud project.
  • Colab requires explicit user authentication.
  • The notebook creates a Cloud Storage bucket with gsutil for staging.
  • Runtime restart is required after installing packages.
  • Agent deployment on Vertex AI Agent Engine is noted as taking about 10 minutes.
  • Tool trajectory evaluation requires returned intermediate steps from the LangChain agent.
  • BYOD evaluation requires predicted_trajectory and response columns instead of a runnable agent.

Best practices

  • Evaluate agents with both objective metrics and subjective feedback to build trust in behavior.
  • Use agent_executor_kwargs={“return_intermediate_steps”: True} so tool calls can be evaluated.
  • Prepare evaluation data with prompts and reference trajectories for tool and trajectory metrics.
  • Evaluate single tool selection before broader trajectory and response quality metrics.
  • Use custom pointwise metrics when response quality depends on whether the response follows tool choices.
  • Clean up experiments and remote agents after running the tutorial.

Graph View

Backlinks