gemini-evaluation-evaluate-with-your-python-code

Evaluate Generative Model Tool Use with Custom Code Execution

Source notebook

Repo path: gemini/evaluation/evaluate_with_your_python_code.ipynb · Open on GitHub · intermediate

Evaluates Gemini tool-use outputs with Vertex AI remote custom metrics.

Summary

This notebook teaches how to use Vertex AI Gen AI Evaluation Service remote custom functions to score tool-use quality. It defines Python metric functions for tool-call validity, tool-name matching, parameter-key matching, and parameter-value matching, then evaluates a pandas dataset. It also computes precision, recall, and F1 from remote custom true-positive, false-positive, and false-negative metrics.

Key code patterns

Initialize Vertex AI evaluation client

import vertexai
 
vertexai.init(project=PROJECT_ID, location=LOCATION)
client = vertexai.Client(project=PROJECT_ID, location=LOCATION)

Sets the project and region used by the evaluation service client.

Define remote custom metric

tool_call_valid_metric = types.Metric(
    name="tool_call_valid",
    remote_custom_function=tool_call_valid_code,
)

Registers Python code that Vertex AI runs remotely to score each evaluation instance.

Create evaluation dataset

dataset = pd.DataFrame({"prompt": prompts,
                        "reference": references,
                        "response": responses})
eval_dataset = types.EvaluationDataset(eval_dataset_df=dataset)

Wraps prompt, reference, and response columns for evaluation.

Run custom metric evaluation

eval_result = client.evals.evaluate(
    dataset=eval_dataset,
    metrics=[tool_call_valid_metric,
             tool_name_match_metric,
             tool_parameter_key_match_metric,
             tool_parameter_kv_match_metric],
)
eval_result.show()

Executes multiple custom tool-use metrics against the dataset.

Compute precision, recall, and F1

precision = mean_tp / (mean_tp + mean_fp) if (mean_tp + mean_fp) > 0 else 0.0
recall = mean_tp / (mean_tp + mean_fn) if (mean_tp + mean_fn) > 0 else 0.0
f1_score = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0

Combines remote custom classification metrics into aggregate tool-call quality scores.

Models & APIs used

• APIs / services: Vertex AI, Vertex Gen AI Evaluation Service
• SDKs / libraries: vertexai, google-cloud-aiplatform[evaluation], pandas

When to use this

Use this pattern when evaluating whether generated responses chose the right tool names and arguments against reference tool calls.

Gotchas & caveats

• Requires google-cloud-aiplatform[evaluation] to be installed or upgraded.
• Colab authentication is needed when running in Google Colab.
• PROJECT_ID must be set before initializing Vertex AI.
• Notebook uses us-central1 as the location.
• Remote custom functions parse JSON from nested gemini_contents response and reference fields.
• Example metrics inspect only the first tool call in each response and reference.

Best practices

• Define separate metrics for tool-call validity, tool-name match, parameter-key match, and parameter key-value match.
• Handle negative examples where no tool calls are expected.
• Return 0.0 for missing expected tool calls and 1.0 for true negatives.
• Use remote custom functions to compute TP, FP, and FN before calculating precision, recall, and F1.

Related

• Concepts: Evaluation · Function Calling & Tools
• Entities: Vertex AI · Vertex AI SDK · Gen AI Evaluation Service · Function Calling · Gemini
• Area: Gemini Notebooks
• Best practices: Evaluation - Best Practices · Function Calling & Tools - Best Practices