Generative AI Google Brain

gemini-evaluation-evaltask-approach-evaluate-gemini-tool-use

3 min read

Evaluate Generative Model Tool Use

Source notebook

Repo path: gemini/evaluation/evaltask_approach/evaluate_gemini_tool_use.ipynb · Open on GitHub · intermediate

Evaluates Gemini function calling and saved tool-call predictions with Vertex AI EvalTask metrics.

Summary

This notebook teaches how to evaluate generative model tool use quality with Vertex AI Python SDK for Gen AI Evaluation Service. It first evaluates a bring-your-own-prediction dataset of saved tool call responses against references, then defines a book_tickets function tool, calls gemini-2.5-flash with google-genai, converts the function call response into the expected JSON shape, and evaluates it with tool-use metrics.

Key code patterns

Initialize Vertex AI

PROJECT_ID = "[your-project-id]"
LOCATION = "us-central1"
 
import vertexai
vertexai.init(project=PROJECT_ID, location=LOCATION)

Sets the project and region used by the Vertex AI SDK and evaluation runs.

Define tool metrics

tool_metrics = [
    "tool_call_valid",
    "tool_name_match",
    "tool_parameter_key_match",
    "tool_parameter_kv_match",
]

Chooses built-in metrics for validating tool call structure, tool name, argument keys, and key-value matches.

Evaluate saved predictions

eval_dataset = pd.DataFrame({
    "response": response,
    "reference": reference,
})
 
tool_use_eval_task = EvalTask(
    dataset=eval_dataset,
    metrics=tool_metrics,
    experiment=experiment_name,
)
eval_result = tool_use_eval_task.evaluate(...)

Shows bring-your-own-prediction evaluation when model responses and expected references are already saved.

Declare a function tool

book_tickets_func = FunctionDeclaration(
    name="book_tickets",
    description="Book movie tickets",
    parameters={"type": "object", "properties": {...}, "required": [...]},
)
book_tickets_tool = Tool(function_declarations=[book_tickets_func])

Defines the function schema Gemini can call, including argument properties and required fields.

Generate and unpack function call

client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)
gemini_response = client.models.generate_content(
    model="gemini-2.5-flash",
    contents=prompt,
    config=types.GenerateContentConfig(tools=[book_tickets_tool]),
)
fc = gemini_response.candidates[0].content.parts[0].function_call

Calls Gemini with a registered tool and extracts the returned function_call for evaluation.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI, Gen AI Evaluation Service
  • SDKs / libraries: google-cloud-aiplatform[evaluation], google-genai, vertexai, pandas

When to use this

Use this pattern when you need objective metrics for Gemini function-calling responses against expected tool-call references.

Gotchas & caveats

  • Requires PROJECT_ID to be set before Vertex AI initialization.
  • Notebook installs google-cloud-aiplatform[evaluation] and google-genai, then restarts the runtime.
  • Colab authentication is only performed when google.colab is detected.
  • The notebook uses LOCATION=“us-central1”.
  • Evaluation expects response and reference columns containing JSON strings with content and tool_calls.

Best practices

  • Define explicit tool evaluation metrics before creating EvalTask.
  • Use a pandas DataFrame with response and reference columns for evaluation datasets.
  • Give FunctionDeclaration parameters clear types, descriptions, and required fields.
  • Use notebook_utils.generate_uuid to create distinct experiment run names.
  • Display evaluation results with notebook_utils.display_eval_result.

Graph View

Backlinks