Generative AI Google Brain

Multimodal Live API

5 min read

Multimodal Live API

Multimodal Live API on Google Cloud is a Gemini realtime, bidirectional streaming interface for text, audio, video frames, tools, and native audio over WebSocket or the Google Gen AI SDK. It is commonly used through Vertex AI and can be deployed into Agent Engine or Agent Runtime for serverless conversational agents.

Synthesized across 10 notebooks. See the source pages under Notebooks below for line-level grounding.

Engineers use the Live API when they need low-latency, turn-aware interaction rather than batch request/response generation. The notebooks show two main access paths: the Google Gen AI SDK for simpler session management, and raw WebSockets when direct control over setup messages, payloads, audio buffers, and interruption handling is required.

A typical session starts with Google Cloud authentication, a project, a region, and a setup/config message that declares modalities, tools, transcription, speech settings, and realtime input behavior. After setup, the client streams client_content for discrete turns or realtime_input for high-frequency audio and video chunks while concurrently receiving model turns, inline audio, transcriptions, tool calls, and turn-completion signals.

Production-oriented examples extend the Live API with Agent Engine, ADK, RAG, Vertex AI Search, Vertex AI Vector Search, and structured function responses. The recurring engineering concerns are authentication freshness, region-specific endpoints, raw PCM audio formats, tool declaration timing, quota-aware retry behavior, session lifecycle, and cleanup of deployed resources.

Canonical workflow

  1. Enable the Vertex AI API, authenticate with Google Cloud credentials, and set PROJECT_ID plus LOCATION or the equivalent environment variables.
  2. Choose the client path: Google Gen AI SDK for simpler Live API sessions, raw WebSocket for direct protocol control, or Agent Engine/Agent Runtime for deployed bidirectional agents.
  3. Create the Live API configuration or setup message with model, response modalities, speech settings, transcription, realtime input settings, system instructions, and any tools required for the session.
  4. Open the Live API session or WebSocket connection, send setup once immediately, and wait for the setup response before streaming user input.
  5. Run send and receive loops concurrently, using client_content for discrete text turns and realtime_input for streaming audio or video frames.
  6. Encode input audio as raw 16-bit PCM at 16 kHz and decode output audio as raw 16-bit PCM at 24 kHz; buffer streamed inline audio until turn completion or interruption handling requires clearing it.
  7. For tools or RAG, retrieve or execute external data in application code, return function results in the expected response format, and ground answers with document chunks and citations where applicable.
  8. Close or clean up sessions, deployed agents, staging resources, and endpoints when finished to avoid stale state or ongoing charges.

Best practices

See Multimodal Live API - Best Practices for the full distilled guide.

  • Use the Google Gen AI SDK when you want simplified Live API session handling; use raw WebSockets when you need full control over handshake, JSON payloads, and low-level streaming behavior.
  • Set PROJECT_ID and LOCATION before constructing model paths, clients, staging buckets, or WebSocket hosts, and use regional aiplatform.googleapis.com endpoints for non-global locations.
  • Refresh google.auth credentials before HTTP or WebSocket calls, especially because application-default access tokens have a limited lifetime.
  • Send the setup message once immediately after opening a WebSocket, wait for the setup response, and declare all tools at session start.
  • Use concurrent send and receive loops, commonly with asyncio.gather or queues, for continuous bidirectional sessions.
  • Use realtime_input for high-frequency audio and video chunks, and client_content for discrete text turns.

Key models & APIs

  • gemini-live-2.5-flash-native-audio, gemini-2.0-flash-live-preview-04-09, Gemini 3.1 Pro, gemini-3.5-flash, gemini-3.1-flash-image, text-embedding-005, Gemini Live API, Multimodal Live API, Google Gen AI SDK, Vertex AI API, Vertex AI Agent Engine, Agent Runtime, ADK, AdkApp, LiveConnectConfig, ProactivityConfig, ToolContext, Vertex AI Search, Vertex AI Vector Search

Notebooks (10)

Agents & ADK (2)

Gemini (8)

Part of the Generative AI Google Brain · Multimodal Live API - Best Practices

Graph View

Backlinks