Generative AI Google Brain

gemini-function-calling-multimodal-function-calling

3 min read

Multimodal Function Calling with the Gemini API & Python SDK

Source notebook

Repo path: gemini/function-calling/multimodal_function_calling.ipynb · Open on GitHub · intermediate

Shows Gemini multimodal function calling with images, video, audio, PDFs, and chat using Google Gen AI SDK.

Summary

This notebook teaches how to define function declarations, package them as tools, and send multimodal inputs to Gemini through Vertex AI. It demonstrates image, video, audio, and PDF function-call prediction, then extracts function names and arguments from responses. The image example executes a Wikipedia lookup and returns the tool response to Gemini for a final natural-language answer, while the chat example shows repeated image-based function-call predictions.

Key code patterns

Initialize Vertex AI client

from google import genai
 
client = genai.Client(
    vertexai=True,
    project=PROJECT_ID,
    location="global",
)

Connects the Google Gen AI SDK to Vertex AI for Gemini requests.

Declare a callable function

get_wildlife_region = FunctionDeclaration(
    name="get_wildlife_region",
    description="Look up the region where an animal can be found",
    parameters={"type": "object", "properties": {"animal": {"type": "string"}}},
)
image_tool = Tool(function_declarations=[get_wildlife_region])

Defines the schema Gemini must use when predicting function calls.

Call Gemini with multimodal input

response = client.models.generate_content(
    model=MODEL_ID,
    contents=[
        Part.from_uri(file_uri="gs://.../multi-color-bird.jpg", mime_type="image/jpeg"),
        "What is the typical habitat or region where this animal lives?",
    ],
    config=GenerateContentConfig(temperature=0, tools=[image_tool]),
)

Shows single-call multimodal function-call prediction from an image and prompt.

Extract function call arguments

function_name = response.function_calls[0].name
function_args = dict(response.function_calls[0].args.items())

Turns Gemini’s structured function-call prediction into executable application data.

Return tool response to Gemini

Content(
    role="tool",
    parts=[Part.from_function_response(
        name=function_name,
        response={"content": api_response},
    )],
)

Feeds external API results back to Gemini so it can generate a final answer.

Create multimodal chat with tools

chat = client.chats.create(
    model=MODEL_ID,
    config=GenerateContentConfig(temperature=0, tools=[chat_tool]),
)
response = chat.send_message([Part.from_uri(file_uri="gs://.../baby-fox.jpg", mime_type="image/jpeg"), prompt])

Demonstrates tool-enabled chat turns grounded in repeated image inputs.

Models & APIs used

  • Models: gemini-3.5-flash
  • APIs / services: Vertex AI, Gemini API, Cloud Storage
  • SDKs / libraries: google-genai, wikipedia

When to use this

Use this pattern when Gemini must inspect media inputs and produce structured function calls for downstream APIs or systems.

Gotchas & caveats

  • Requires a Google Cloud project with the Vertex AI API enabled.
  • Colab requires explicit auth.authenticate_user(); Vertex AI Workbench does not require that Colab-only step.
  • The notebook uses billable Vertex AI components.
  • PROJECT_ID must be set directly or via GOOGLE_CLOUD_PROJECT.
  • LOCATION is set to global.
  • Media files are loaded from gs:// Cloud Storage URIs with explicit MIME types.
  • Only the image example executes an external API call and returns the result to Gemini; other examples stop at predicted calls.

Best practices

  • Use FunctionDeclaration parameters to constrain predicted function arguments to a JSON schema.
  • Group function declarations into Tool objects before passing them to Gemini.
  • Set temperature=0 for deterministic function-call prediction examples.
  • Use Part.from_uri with the correct MIME type for image, video, audio, and PDF inputs.
  • Save the model response content when returning a later tool response with the thought signature.
  • Send tool results back as Content(role=“tool”) with Part.from_function_response.

Graph View

Backlinks