Generative AI Google Brain

gemini-multimodal-live-api-intro-multimodal-live-api

4 min read

Getting Started with Gemini Live API using WebSocket

Source notebook

Repo path: gemini/multimodal-live-api/intro_multimodal_live_api.ipynb · Open on GitHub · intermediate

Uses Gemini Live API over WebSocket for text, audio, video, tools, and native audio features.

Summary

The notebook teaches how to create a low-level WebSocket session to the Gemini Live API in Vertex AI with OAuth bearer authentication, setup messages, and concurrent send and receive loops. It demonstrates text-to-audio, audio-to-audio, video-to-audio, function calling, Google Search, audio transcription, VAD, proactive audio, and affective dialog. The workflow sets project, location, and model, sends setup and client or realtime input payloads, decodes Base64 PCM audio responses, and exits turns on turnComplete.

Key code patterns

Build Live API WebSocket URL

api_host = "aiplatform.googleapis.com"
if LOCATION != "global":
    api_host = f"{LOCATION}-aiplatform.googleapis.com"
service_url = f"wss://{api_host}/ws/google.cloud.aiplatform.v1.LlmBidiService/BidiGenerateContent"
headers = {"Authorization": f"Bearer {token_list[0]}"}

Shows the regional Vertex AI WebSocket endpoint and OAuth bearer-token authentication.

Send setup first

setup = {
    "setup": {
        "model": model,
        "system_instruction": system_instruction,
        "generation_config": config,
    }
}
await ws.send(json.dumps(setup))
await ws.recv()

The session must be initialized with model and generation config before streaming starts.

Concurrent send and receive loops

async def send_loop():
    await asyncio.sleep(0.02)
 
async def receive_loop():
    async for message in ws:
        print("Received message")
 
await asyncio.gather(send_loop(), receive_loop())

Bidirectional Live API clients send input and receive model output at the same time.

Text turn to audio output

msg = {
    "client_content": {
        "turns": [{"role": "user", "parts": [{"text": text_input}]}],
        "turn_complete": True,
    }
}
await ws.send(json.dumps(msg))

Uses client_content for a discrete text turn that immediately requests a response.

Stream audio chunks

msg = {
    "realtime_input": {
        "media_chunks": [{
            "mime_type": "audio/pcm;rate=16000",
            "data": base64.b64encode(chunk).decode("utf-8"),
        }]
    }
}
await ws.send(json.dumps(msg))

Uses realtime_input with Base64 PCM chunks for low-latency audio streaming.

Stream video frames

processed_jpeg = encode_image(video_data, DEFAULT_IMAGE_ENCODE_OPTIONS)
b64_data = base64.b64encode(processed_jpeg).decode("utf-8")
msg = {
    "realtime_input": {
        "video": {"mime_type": "image/jpeg", "data": b64_data}
    }
}
await ws.send(json.dumps(msg))

The Live API expects video as discrete JPEG image frames, not a continuous MP4 stream.

Declare tools in setup

get_temperature_declaration = {
    "name": "get_temperature",
    "description": "Gets the current temperature for a given location.",
    "parameters": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]},
}
tools = {"function_declarations": [get_temperature_declaration]}
setup = {"setup": {"model": model, "generation_config": config, "tools": tools}}

Function declarations must be sent at session start as part of the setup message.

Models & APIs used

  • Models: gemini-live-2.5-flash-native-audio
  • APIs / services: Vertex AI, Gemini Live API, Cloud Storage, Google Search
  • SDKs / libraries: websockets, opencv-python, numpy, IPython.display, google.colab

When to use this

Use this pattern when you need a custom low-level Gemini Live API client with real-time text, audio, video, and tool interactions over WebSocket.

Gotchas & caveats

  • A Google Cloud project is required and the Vertex AI API must be enabled.
  • The notebook gets an Application Default Credentials token with gcloud and notes the default access token lifetime is 3600 seconds.
  • LOCATION changes the WebSocket host and model resource path; the sample uses us-central1 unless global is selected.
  • The setup message must be sent immediately after the WebSocket connection is established.
  • Input audio must be raw 16-bit PCM at 16kHz little-endian; output audio is raw 16-bit PCM at 24kHz little-endian.
  • Video input is sent as image frames; the notebook states Live API supports frames at 1FPS and recommends native 768x768 at 1FPS.
  • All functions must be declared at the start of the session in the setup message.
  • Clients need to handle interrupted messages for barge-in and send audioStreamEnd when audio pauses for more than a second.

Best practices

  • Set PROJECT_ID and LOCATION before constructing the model path.
  • Use the regional aiplatform.googleapis.com WebSocket endpoint for non-global locations.
  • Send setup once, await the setup response, then start streaming.
  • Run send_loop and receive_loop concurrently with asyncio.gather for bidirectional interaction.
  • Use realtime_input for high-frequency audio and video chunks and client_content for discrete text turns.
  • Decode inlineData audio from serverContent modelTurn parts and play it at 24000 Hz.
  • Break a demo turn when serverContent.turnComplete is received.
  • Use timeout and connection-closed handling in reusable response helpers.

Graph View

Backlinks