Generative AI Google Brain

partner-models-claude-claude-multimodal-function-calling

3 min read

Multimodal Function Calling with Claude Models

Source notebook

Repo path: partner-models/claude/claude_multimodal_function_calling.ipynb · Open on GitHub · intermediate

Uses Claude on Vertex AI for multimodal prompting and tool use with a Wikipedia lookup.

Summary

This notebook shows how to call Anthropic Claude models through Vertex AI using the AnthropicVertex client. It builds a multimodal message with a base64-encoded image and text, then adds a tool definition so Claude can request an external wildlife-region lookup. The workflow demonstrates the four-step tool loop: define the tool, inspect Claude’s ToolUseBlock, call Wikipedia in developer code, and send a tool_result back for a final conversational answer.

Key code patterns

Create Anthropic Vertex client

from anthropic import AnthropicVertex
 
client = AnthropicVertex(region=LOCATION, project_id=PROJECT_ID)

Initializes Claude access through the Vertex AI API endpoint for the selected project and region.

Build multimodal message

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "source": {"type": "base64", "media_type": image_media_type, "data": image_b64}},
        {"type": "text", "text": "What is the region where this animal lives?"},
    ],
}]

Shows Claude message format for combining an image and text in one user turn.

Send tool definition

TOOL = {
    "name": "get_wildlife_region",
    "description": "Look up the region where an animal can be found",
    "input_schema": {"type": "object", "properties": {"animal": {"type": "string"}}},
}
response = client.messages.create(max_tokens=1024, messages=messages, tools=[TOOL], model=MODEL)

Lets Claude decide whether to request a structured tool call for habitat information.

Return tool result

animal_name = response.content[1].input["animal"]
api_response = wikipedia.page(animal_name).content
messages.append({"role": "assistant", "content": response.content})
messages.append({"role": "user", "content": [{"type": "tool_result", "tool_use_id": tool_use.id, "content": api_response}]})

Implements the developer-executed API call and passes the result back to Claude.

Models & APIs used

  • Models: claude-3-5-sonnet-v2@20241022
  • APIs / services: Vertex AI, Wikipedia API
  • SDKs / libraries: anthropic[vertex], wikipedia, httpx, requests, ipywidgets

When to use this

Use this pattern when a Claude multimodal workflow needs grounded external data retrieved through developer-controlled tools.

Gotchas & caveats

  • Requires enabling the Vertex AI API in a Google Cloud project.
  • Claude models must be enabled in Vertex AI Model Garden and terms of service accepted before prompting.
  • Model availability depends on region; the notebook maps selected Claude models to allowed regions.
  • Colab requires explicit user authentication.
  • Images must be base64 encoded with the correct media_type before being sent in the message.
  • The developer must execute the external API call; Claude only returns the tool request.

Best practices

  • Select a Claude model before creating the AnthropicVertex client because models have different location availability.
  • Use environment variables GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION when explicit project or region values are not provided.
  • Define tools with a name, description, and input_schema.
  • Append the assistant tool-use response and a user tool_result message before asking Claude for the final answer.
  • Pass tools again in the follow-up messages.create call.

Graph View

Backlinks