Generative AI Google Brain

gemini-prompts-prompt-optimizer-get-started-with-vertex-ai-prompt-optimizer-tool-usage

3 min read

Get Started with Vertex AI Prompt Optimizer - Tool usage

Source notebook

Repo path: gemini/prompts/prompt_optimizer/get_started_with_vertex_ai_prompt_optimizer_tool_usage.ipynb · Open on GitHub · intermediate

Optimizes a Gemini tool-calling system instruction with Vertex AI Prompt Optimizer data-driven mode.

Summary

This notebook teaches how to use Vertex AI Prompt Optimizer to improve tool usage for a Gemini model. It prepares Google Cloud project settings, IAM permissions, Cloud Storage paths, function declarations, tool config JSON, and an optimization configuration. The workflow runs a Prompt Optimizer job with tool-call evaluation metrics, then retrieves the best optimized instruction from GCS results.

Key code patterns

Initialize Vertex AI client

import vertexai
 
client = vertexai.Client(
    project=PROJECT_ID,
    location=LOCATION,
)

Creates the client used to submit the Prompt Optimizer job.

Declare tools

get_company_information = FunctionDeclaration(
    name="get_company_information",
    description="Retrieves financial performance to provide an overview for a company.",
    parameters={"type": "object", "properties": {"ticker": {"type": "string"}}, "required": ["ticker"]},
)

Defines function calling tools with OpenAPI-compatible schemas.

Restrict tool calls

tool_config = ToolConfig(
    function_calling_config=ToolConfig.FunctionCallingConfig(
        mode=ToolConfig.FunctionCallingConfig.Mode.ANY,
        allowed_function_names=["get_company_information", "get_stock_price", "get_company_news", "get_company_sentiment"],
    )
)

Controls which functions the Gemini model may call during optimization.

Build optimizer config

vapo_data_settings = {
    "target_model": "gemini-2.5-flash",
    "optimization_mode": "instruction",
    "tools": vapo_tools,
    "tool_config": vapo_tool_config,
    "eval_metrics_types": ["tool_name_match", "tool_parameter_key_match", "tool_parameter_kv_match"],
}

Configures data-driven prompt optimization for tool-call accuracy.

Run optimization job

result = client.prompt_optimizer.optimize(
    method="vapo",
    config={
        "config_path": config_path,
        "wait_for_completion": True,
        "service_account": SERVICE_ACCOUNT,
    },
)

Starts the Vertex AI backend Prompt Optimizer custom job.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI, Cloud Storage, Artifact Registry, Gen AI Evaluation service
  • SDKs / libraries: google-cloud-aiplatform, vertexai, pydantic, etils, google.cloud.storage, jsonschema, pandas

When to use this

Use this pattern when you have tool-call examples and want Vertex AI to optimize a Gemini system instruction for valid function calls.

Gotchas & caveats

  • Requires Vertex AI API enabled for the Google Cloud project.
  • Requires a Cloud Storage bucket for input data, config, and results.
  • Default Compute Engine service account needs Vertex AI User, Storage Object Admin, and Artifact Registry Reader roles.
  • The notebook uses us-central1 as the default location.
  • target_model supports gemini-2.5-flash and gemini-2.5-pro in the shown config.
  • num_steps must be between 10 and 20 in the pydantic config.
  • The example stock price function declares ticker as integer while other ticker parameters use string.
  • The dataset target must be a JSON string aligned with Gen AI Evaluation service tool-use evaluation.

Best practices

  • Use a structured pydantic OptimizationConfig before submitting the job.
  • Validate tools and tool_config JSON before running optimization.
  • Pass FunctionDeclaration and ToolConfig as JSON structures to Prompt Optimizer.
  • Use weighted tool-call metrics for tool name, parameter key, and parameter key-value matching.
  • Retrieve the best prompt programmatically from GCS output files for application use.
  • Clean up the custom job and Cloud Storage bucket after the tutorial.

Graph View

Backlinks