Generative AI Google Brain

gemini-agent-engine-evaluating-crewai-agent-engine-customized-template

3 min read

Evaluate a CrewAI agent on Vertex AI Agent Engine (Customized template)

Source notebook

Repo path: gemini/agent-engine/evaluating_crewai_agent_engine_customized_template.ipynb · Open on GitHub · advanced

Evaluates a CrewAI Gemini agent on Vertex AI Agent Engine with tool, trajectory, and response metrics.

Summary

This notebook builds a CrewAI product support agent that uses custom product detail and price tools with a Gemini model. It deploys the custom app to Vertex AI Agent Engine, prepares an evaluation dataset with prompts and reference trajectories, and runs Vertex AI Gen AI Evaluation for single-tool use, trajectory matching, response quality, and a custom pointwise metric. It also shows a BYOD-style evaluation flow and cleanup of the experiment and remote agent.

Key code patterns

Initialize Vertex AI

vertexai.init(
    project=PROJECT_ID,
    location=LOCATION,
    staging_bucket=BUCKET_URI,
    experiment=EXPERIMENT_NAME,
)

Sets project, region, staging bucket, and experiment context before deploying and evaluating agents.

CrewAI Custom App

class CrewAIApp:
    def query(self, input: str):
        product_researcher = Agent(
            role="Product Researcher",
            llm=model,
            tools=[get_product_details, get_product_price],
        )
        crew = Crew(agents=[product_researcher], tasks=[research_task])
        return parse_crewai_output_to_dictionary(crew, crew.kickoff())

Wraps a CrewAI workflow in the Agent Engine custom template shape with setup and query methods.

Deploy Agent Engine App

remote_custom_agent = agent_engines.create(
    local_custom_agent,
    requirements=[
        "google-cloud-aiplatform[agent_engines]",
        "crewai",
        "crewai-tools",
    ],
)

Packages the local CrewAI app and dependencies for remote execution on Vertex AI Agent Engine.

Trajectory Evaluation Dataset

eval_sample_dataset = pd.DataFrame({
    "prompt": [...],
    "reference_trajectory": [[{
        "tool_name": "get_product_price",
        "tool_input": {"product_name": "smartphone"},
    }]],
})

Provides prompts and expected tool-call trajectories required by trajectory metrics.

Run EvalTask

trajectory_eval_task = EvalTask(
    dataset=eval_sample_dataset,
    metrics=trajectory_metrics,
    experiment=EXPERIMENT_NAME,
)
trajectory_eval_result = trajectory_eval_task.evaluate(
    runnable=agent_parsed_response
)

Runs Vertex AI Gen AI Evaluation against a callable remote agent wrapper.

Custom Pointwise Metric

response_follows_trajectory_metric = PointwiseMetric(
    metric="response_follows_trajectory",
    metric_prompt_template=response_follows_trajectory_prompt_template,
)

Adds a model-based metric that judges whether the response follows the predicted trajectory.

Models & APIs used

  • Models: vertex_ai/gemini-2.0-flash
  • APIs / services: Vertex AI, Vertex AI Gen AI Evaluation, Vertex AI Agent Engine, Cloud Storage
  • SDKs / libraries: google-cloud-aiplatform, vertexai, crewai, crewai-tools, pandas, plotly

When to use this

Use this pattern when evaluating a deployed custom CrewAI agent on Agent Engine against expected tool usage, trajectories, and response quality.

Gotchas & caveats

  • Requires an existing Google Cloud project with the Vertex AI API enabled.
  • Colab requires explicit user authentication with google.colab.auth.authenticate_user().
  • The notebook creates a Cloud Storage bucket with gsutil mb for the Vertex AI staging bucket.
  • The runtime must be restarted after installing google-cloud-aiplatform, crewai, crewai-tools, cloudpickle, pydantic, and requests.
  • Agent deployment on Vertex AI Agent Engine is noted as requiring about 10 minutes.
  • The BYOD section defines byod_eval_data but creates byod_eval_sample_dataset from eval_data.

Best practices

  • Evaluate agents both online and offline, using subjective and objective evaluation signals.
  • Track single tool selection, trajectory order, response generation, latency, and failure rate when evaluating agents.
  • Use reference trajectories in the evaluation dataset when checking expected tool calls.
  • Wrap custom agent output so it includes response and predicted_trajectory for evaluation.
  • Clean up experiments and remote agents after the notebook run.

Graph View

Backlinks