Generative AI Google Brain

Multimodal Live API - Best Practices

6 min read

Multimodal Live API — Best Practices

Distilled from 10 notebooks tagged Multimodal Live API in the GoogleCloudPlatform/generative-ai repository. The From the notebooks section below cites the per-notebook source for grounding.

Do this

  • Use the Google Gen AI SDK when you want simplified Live API session handling; use raw WebSockets when you need full control over handshake, JSON payloads, and low-level streaming behavior.
  • Set PROJECT_ID and LOCATION before constructing model paths, clients, staging buckets, or WebSocket hosts, and use regional aiplatform.googleapis.com endpoints for non-global locations.
  • Refresh google.auth credentials before HTTP or WebSocket calls, especially because application-default access tokens have a limited lifetime.
  • Send the setup message once immediately after opening a WebSocket, wait for the setup response, and declare all tools at session start.
  • Use concurrent send and receive loops, commonly with asyncio.gather or queues, for continuous bidirectional sessions.
  • Use realtime_input for high-frequency audio and video chunks, and client_content for discrete text turns.
  • Set end_of_turn to true when accumulated client content should trigger generation, and use turn_complete or serverContent.turnComplete to stop collecting a response.
  • Send audio_stream_end or audioStreamEnd when realtime audio pauses for more than about a second.
  • Use the required audio formats consistently: input as raw 16-bit PCM at 16 kHz little-endian, output as raw 16-bit PCM at 24 kHz little-endian, and inline audio MIME types such as audio/pcm or audio/pcm;rate=16000 where shown.
  • Decode inline audio from serverContent modelTurn parts, concatenate int16 PCM chunks, and clear playback buffers when interruption or barge-in occurs.
  • Use system instructions to define agent role, response guidance, proactive behavior, persona, and formatting expectations.
  • For Agent Engine, keep init lightweight and pickle-able, move heavy initialization to set_up, list runtime dependencies explicitly, and wrap ADK agents in AdkApp before deployment.
  • For Agent Runtime or Agent Engine deployments, configure resource limits, max instances, staging buckets, source packages, and cleanup routines deliberately.
  • For RAG scenarios, ground answers in retrieved chunks, keep document metadata such as page and chunk number, use the same embedding model for chunks and queries, and add retry with exponential backoff for quota-sensitive calls.
  • Verify citations, be careful with trusted versus untrusted uploaded files, and avoid combining sensitive internal data with public browsing without review.

Avoid this

  • Starting without an enabled Vertex AI API, authenticated Google Cloud user, project ID, or required region setting.
  • Using the wrong regional WebSocket host or model resource path when LOCATION changes between global and us-central1.
  • Streaming before sending setup or declaring tools after the session has already started.
  • Sending audio in the wrong format, sample rate, endian order, MIME type, or assuming output audio uses the same rate as input audio.
  • Forgetting that Live API sessions are single WebSocket connections and that session context is erased when the connection terminates.
  • Not handling interruptions, connection closures, timeouts, turn completion, or audio stream end signals in the client loop.
  • Omitting required Agent Engine runtime dependencies, using heavy non-serializable initialization in init, or failing to create a session before run_live.
  • Leaving deployed agents, staging buckets, Vector Search endpoints, or other managed resources running after a tutorial, causing avoidable charges.

From the notebooks

Getting Started with Bidirectional Streaming v2 on Agent Runtime

  • Use bring-your-own-Dockerfile deployment with source_packages for custom Agent Runtime servers.
  • Set GOOGLE_GENAI_USE_VERTEXAI to 1 for the deployed ADK agent environment.
  • Use resource_limits and max_instances in Agent Runtime deployment config.
  • Refresh google.auth credentials before making HTTP or WebSocket calls.
  • Separate receive and send loops for bidirectional WebSocket handling.

Getting Started with Live API on Agent Engine

  • Keep init lightweight and pickle-able for Agent Engine serialization.
  • Put heavy initialization in set_up because Agent Engine calls it when the serverless container starts.
  • Make each stream_query yield a complete serializable response object.
  • Use bidi_stream_query with asyncio.Queue for continuous two-way sessions.
  • Declare agent dependencies in the Agent Engine requirements config.

Introduction to Gemini Deep Research Agent

  • Save the interaction_id immediately after initialization.
  • Use specific formatting instructions in the prompt to shape reports, sections, tables, and tone.
  • Prompt the agent to state when data is unavailable instead of estimating it.
  • Be cautious when combining sensitive internal data with public web browsing.
  • Verify citations returned by the agent.

Getting Started with the Live API Native Audio

  • Use a reusable configuration helper for transcription, proactivity, affective dialog, and system instructions.
  • Use an async context manager for the Live API session lifecycle.
  • Send conversation turns sequentially to maintain session context.
  • Collect input and output transcriptions alongside streamed audio when transcription is enabled.
  • Use system instructions to constrain proactive chime-in behavior and set response persona.

Getting Started with Gemini Live API using WebSocket

  • Set PROJECT_ID and LOCATION before constructing the model path.
  • Use the regional aiplatform.googleapis.com WebSocket endpoint for non-global locations.
  • Send setup once, await the setup response, then start streaming.
  • Run send_loop and receive_loop concurrently with asyncio.gather for bidirectional interaction.
  • Use realtime_input for high-frequency audio and video chunks and client_content for discrete text turns.

Getting Started with Gemini Live API using Gen AI SDK

  • Use LiveConnectConfig to centralize response modalities, speech configuration, tools, transcription, and realtime input settings.
  • Set end_of_turn to True when generation should start after accumulated client content.
  • Declare tools when initiating the session rather than after the session starts.
  • Use turn_complete to know when to stop collecting an audio response in conversational loops.
  • Send audio_stream_end=True when a realtime audio stream is paused for more than a second.

Gemini Live API Quickstart

  • Use the Google Gen AI SDK for a simplified Live API session and interruption handling.
  • Use WebSockets when you need direct control over the handshake and raw JSON payloads.
  • Stream audio in small chunks and delay briefly to simulate real-time input from a microphone.
  • Set the input MIME type to audio/pcm;rate=16000 when sending audio.
  • Decode inline audio data and concatenate int16 PCM chunks before playback.

Interactive Loan Application Assistant (Financial Services)

  • Ground generated answers in retrieved document chunks instead of asking the model without context.
  • Include page number and chunk number in citations when generating RAG answers.
  • Use retry with exponential backoff around embedding and generation calls for quota management.
  • Skip unreadable, empty, or blank PDF pages during extraction.
  • Use chunking and embeddings for faster targeted retrieval over large loan documents.

Real-time Retrieval Augmented Generation (RAG) using the Multimodal Live API with Gemini 2.0

  • Use genai.Client with vertexai=True, project, and location for Vertex AI endpoint calls.
  • Ground domain-specific answers with retrieved document context instead of relying on pretrained model knowledge.
  • Use the same embedding model for document chunks and user query embeddings.
  • Keep page number, chunk number, and document metadata with retrieved chunks.
  • Add exponential retry handling for quota-sensitive embedding and answer generation calls.

Get hands-on with a customer support use case using Gemini and Gen AI SDK

  • Use system_instruction to give task role and response guidance across the interaction.
  • Use Part.from_uri with explicit MIME types for image inputs.
  • Use response_schema and response_mime_type=“application/json” for downstream structured processing.
  • Use temperature=0 when calling Google Search for store-location lookup.
  • Return function results with Part.from_function_response so the model can incorporate external data.

Back to Multimodal Live API · Best Practices Map

Graph View

Backlinks