Generative AI Google Brain

gemini-evaluation-multi-agent-evals-with-arize-and-crewai

3 min read

Overview

Source notebook

Repo path: gemini/evaluation/multi_agent_evals_with_arize_and_crewai.ipynb · Open on GitHub · intermediate

Evaluates a CrewAI research crew with Phoenix tracing and Vertex AI trajectory metrics.

Summary

This notebook teaches how to build and evaluate a sequential multi-agent CrewAI workflow. It configures API keys, registers Phoenix auto-instrumentation, defines researcher, fact-checker, and writer agents using Gemini, and creates Phoenix datasets for test topics. It then runs experiments with Google Gen AI trajectory evaluators and stores traces and metrics in Phoenix.

Key code patterns

API key setup

serper_key = os.getenv("SERPER_API_KEY", "SERPER_API_KEY")
phoenix_api_key = os.getenv("PHOENIX_API_KEY", "PHOENIX_API_KEY")
gemini_api_key = os.getenv("GEMINI_API_KEY", "GEMINI_API_KEY")
os.environ["SERPER_API_KEY"] = serper_key
os.environ["GEMINI_API_KEY"] = gemini_api_key
os.environ["PHOENIX_API_KEY"] = phoenix_api_key
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "https://app.phoenix.arize.com/"

The workflow depends on Serper search, Gemini model access, and Phoenix trace export.

Phoenix auto-instrumentation

from phoenix.otel import register
 
tracer_provider = register(auto_instrument=True)

Auto-instrumentation captures CrewAI traces when the matching OpenInference package is installed.

Sequential CrewAI agents

search_tool = SerperDevTool()
llm = "gemini-2.0-flash"
crew = Crew(
    agents=[researcher, fact_checker, writer],
    tasks=[conduct_analysis_task, fact_checking_task, writer_task],
    verbose=False,
    process=Process.sequential,
)

The notebook models a fixed agent trajectory from research to verification to writing.

Phoenix dataset creation

phoenix_client = px.Client()
try:
    dataset = phoenix_client.get_dataset(name="crewai-researcher-test-topics")
except ValueError:
    dataset = phoenix_client.upload_dataset(
        dataframe=df,
        dataset_name="crewai-researcher-test-topics",
        input_keys=["topic"],
        output_keys=["reference_trajectory"],
    )

Datasets centralize test inputs and expected trajectories for repeated experiments.

Trajectory evaluator

eval_task = EvalTask(
    dataset=eval_dataset,
    metrics=[metric_name],
)
eval_result = eval_task.evaluate()
metric_value = eval_result.summary_metrics.get(f"{metric_name}/mean")
if metric_value is None:
    return 0.0
return metric_value

Vertex AI evaluation metrics compare predicted and reference agent trajectories.

Experiment run

experiment = run_experiment(
    dataset,
    call_crew_with_topic,
    experiment_name="agent-experiment",
    evaluators=[
        trajectory_exact_match,
        trajectory_precision,
        trajectory_in_order_match,
        trajectory_any_order_match,
        agent_names_match,
    ],
)

Phoenix runs each dataset row through the crew and applies multiple evaluators.

Models & APIs used

  • Models: gemini-2.0-flash
  • APIs / services: Vertex AI, Google Gen AI Evaluation Service, Arize Phoenix, Serper API
  • SDKs / libraries: arize-phoenix, crewai, crewai_tools, openinference-instrumentation-crewai, phoenix, vertexai, pandas, nest_asyncio

When to use this

Use this pattern when evaluating whether a CrewAI multi-agent workflow follows an expected trajectory while collecting traces in Phoenix.

Gotchas & caveats

  • Requires SERPER_API_KEY, PHOENIX_API_KEY, GEMINI_API_KEY, and gcloud auth login.
  • Phoenix hosted tracing requires PHOENIX_COLLECTOR_ENDPOINT and PHOENIX_CLIENT_HEADERS.
  • CrewAI tracing depends on openinference-instrumentation-crewai and register(auto_instrument=True).
  • The trajectory parser only reads response.get(“tasks_output”) and returns an empty trajectory if tasks are missing.
  • EvalTask summary metrics may be missing, and the notebook maps missing metric values to 0.0.
  • The Phoenix dataset is created only when get_dataset raises ValueError.

Best practices

  • Load API keys from environment variables and prompt with getpass when missing.
  • Use explicit task context to enforce sequential dependencies between agents.
  • Store test inputs and expected reference trajectories in a Phoenix dataset.
  • Evaluate trajectories with multiple metrics: exact match, precision, in-order match, and any-order match.
  • Add a custom code evaluator for agent-name order matching.
  • Apply nest_asyncio before running Phoenix experiments in the notebook environment.

Graph View

Backlinks