Generative AI Google Brain

gemini-evaluation-evaltask-approach-evaluating-crewai-agent

3 min read

Evaluating Agents - Evaluate a CrewAI agent with Vertex AI Gen AI Evaluation Service

Source notebook

Repo path: gemini/evaluation/evaltask_approach/evaluating_crewai_agent.ipynb · Open on GitHub · intermediate

Evaluates a CrewAI product agent with Vertex AI Gen AI Evaluation metrics and BYOD evaluation data.

Summary

This notebook builds a local CrewAI customer support-style agent that uses Gemini and two custom product tools for details and prices. It prepares an agent evaluation dataset with prompts and reference trajectories, then runs Vertex AI Gen AI Evaluation for single-tool usage, trajectory metrics, response metrics, and a custom pointwise metric. It also demonstrates a bring-your-own-dataset flow where predicted trajectories and responses are evaluated without rerunning the agent.

Key code patterns

Initialize Vertex AI experiment

PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")
LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "us-central1")
EXPERIMENT_NAME = "evaluate-crewai-agent"
vertexai.init(project=PROJECT_ID, location=LOCATION, experiment=EXPERIMENT_NAME)

Sets project, region, and experiment context before running evaluation tasks.

Define CrewAI tools

@tool
def get_product_details(product_name: str):
    return details.get(product_name, "Product details not found.")
 
@tool
def get_product_price(product_name: str):
    return details.get(product_name, "Product price not found.")

Creates explicit tools whose calls can be captured and compared against reference trajectories.

Wrap agent as runnable

def agent_parsed_outcome(input):
    agent = Agent(llm="vertex_ai/gemini-2.5-flash", tools=[get_product_details, get_product_price])
    task = Task(description=f"Analyze this user request: '{input}'.", agent=agent)
    crew = Crew(agents=[agent], tasks=[task], process=Process.sequential)
    result = crew.kickoff()
    return parse_crewai_output_to_dictionary(crew, result)

Adapts a CrewAI agent into the callable structure expected by EvalTask.evaluate.

Run trajectory evaluation

trajectory_metrics = [
    "trajectory_exact_match",
    "trajectory_in_order_match",
    "trajectory_any_order_match",
    "trajectory_precision",
    "trajectory_recall",
]
EvalTask(dataset=eval_sample_dataset, metrics=trajectory_metrics).evaluate(runnable=agent_parsed_outcome)

Evaluates whether predicted tool sequences match reference tool trajectories.

Create custom response metric

template = PointwiseMetricPromptTemplate(
    criteria=criteria,
    rating_rubric={"1": "Follows trajectory", "0": "Does not follow trajectory"},
    input_variables=["prompt", "predicted_trajectory"],
)
metric = PointwiseMetric(metric="response_follows_trajectory", metric_prompt_template=template)

Shows how to score whether the final response logically follows the agent trajectory.

Evaluate BYOD data

byod_eval_sample_dataset["predicted_trajectory"] = byod_eval_sample_dataset["predicted_trajectory"].apply(json.dumps)
byod_eval_sample_dataset["reference_trajectory"] = byod_eval_sample_dataset["reference_trajectory"].apply(json.dumps)
EvalTask(dataset=byod_eval_sample_dataset, metrics=response_tool_metrics).evaluate()

Supports offline evaluation when predictions and responses are already available.

Models & APIs used

  • Models: vertex_ai/gemini-2.5-flash
  • APIs / services: Vertex AI Gen AI Evaluation, Vertex AI
  • SDKs / libraries: crewai, crewai-tools, google-cloud-aiplatform[evaluation], vertexai, pandas, plotly

When to use this

Use this pattern when you need offline or prototype-stage evaluation of an agent’s tool selection, tool trajectory, and final response quality.

Gotchas & caveats

  • The notebook installs crewai0.95.0 and crewai-tools0.25.8, then restarts the runtime.
  • Colab authentication is required only when running in Google Colab.
  • A Google Cloud project with the Vertex AI API enabled is required.
  • LOCATION defaults to us-central1 from GOOGLE_CLOUD_REGION when not set.
  • Some EvalTask runs use BUCKET_URI for output_uri_prefix, but BUCKET_URI is not defined in the provided notebook text.
  • CrewAI tool call parsing depends on agent.tools_results being present; the helper records an error if that attribute is missing.
  • BYOD trajectory columns are converted with json.dumps before evaluation.

Best practices

  • Evaluate agents with both monitoring-style task metrics and observability considerations such as latency and failure rate.
  • Use reference trajectories to evaluate expected tool choices and ordering.
  • Start with single-tool usage evaluation before broader trajectory evaluation.
  • Evaluate final responses separately from tool trajectory quality.
  • Use custom pointwise metrics when standard text metrics are insufficient for agent behavior.
  • Clean up the Vertex AI experiment when finished by deleting backing TensorBoard runs.

Graph View

Backlinks