Generative AI Google Brain

gemini-multimodal-live-api-live-api-quickstart

3 min read

Gemini Live API Quickstart

Source notebook

Repo path: gemini/multimodal-live-api/live_api_quickstart.ipynb · Open on GitHub · intermediate

Connects to Gemini Live API for bidirectional audio streaming with Gen AI SDK and raw WebSockets.

Summary

This notebook teaches how to establish a Gemini Live API session on Vertex AI and stream audio to a Gemini model. It demonstrates two end-to-end paths: a higher-level Google Gen AI SDK session and a lower-level WebSocket flow using bearer-token authentication, JSON setup payloads, base64 audio chunks, and playback of 24 kHz PCM responses.

Key code patterns

Vertex AI Gen AI client

PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")
LOCATION = "us-central1"
client = genai.Client(
    enterprise=True,
    project=PROJECT_ID,
    location=LOCATION,
)

Configures the Google Gen AI SDK client to use a Google Cloud project and Vertex AI location.

SDK live audio session

async with client.aio.live.connect(model=MODEL_ID, config=config) as session:
    with open("input.wav", "rb") as f:
        while chunk := f.read(1024):
            await session.send_realtime_input(
                audio=types.Blob(data=chunk, mime_type="audio/pcm;rate=16000")
            )
            await asyncio.sleep(0.01)

Shows the simplified SDK pattern for opening a Live API session and streaming PCM audio chunks.

Handle SDK audio responses

async for message in session.receive():
    if message.server_content.interrupted:
        print("[Interrupted] Clear client audio buffer immediately.")
        continue
    for part in message.server_content.model_turn.parts:
        if part.inline_data:
            audio_data.append(np.frombuffer(part.inline_data.data, dtype=np.int16))
    if message.server_content.turn_complete:
        display(Audio(np.concatenate(audio_data), rate=24000, autoplay=True))

Processes streamed model audio, handles interruption signals, and plays the completed response.

Raw WebSocket setup

headers = {"Authorization": f"Bearer {token_list[0]}"}
MODEL = f"projects/{PROJECT_ID}/locations/{LOCATION}/publishers/google/models/{MODEL_ID}"
URI = f"wss://{LOCATION}-aiplatform.googleapis.com/ws/google.cloud.aiplatform.v1.LlmBidiService/BidiGenerateContent"
async with websockets.connect(URI, additional_headers=headers) as ws:
    await ws.send(json.dumps({"setup": {"model": MODEL, "generation_config": config}}))

Demonstrates the low-level handshake required when using standard WebSockets directly.

Base64 WebSocket audio chunks

msg = {
    "realtime_input": {
        "media_chunks": [{
            "mime_type": "audio/pcm;rate=16000",
            "data": base64.b64encode(chunk).decode("utf-8"),
        }]
    }
}
await ws.send(json.dumps(msg))

Shows how raw PCM audio must be base64 encoded inside WebSocket JSON payloads.

Models & APIs used

When to use this

Use this pattern when building a real-time audio application that streams microphone-like input to Gemini and plays streamed audio responses.

Gotchas & caveats

  • Enable the Vertex AI API before running the notebook.
  • The notebook uses a Google Cloud project for authentication, not an API key.
  • PROJECT_ID must be set explicitly or available as GOOGLE_CLOUD_PROJECT.
  • The configured location is us-central1.
  • Input audio must be raw 16-bit PCM at 16 kHz, little-endian.
  • Output audio is raw 16-bit PCM at 24 kHz, little-endian.
  • The WebSocket path requires an application-default access token from gcloud.
  • Clients must decode, buffer, and play streamed audio chunks, and clear playback buffers on interruption.

Best practices

  • Use the Google Gen AI SDK for a simplified Live API session and interruption handling.
  • Use WebSockets when you need direct control over the handshake and raw JSON payloads.
  • Stream audio in small chunks and delay briefly to simulate real-time input from a microphone.
  • Set the input MIME type to audio/pcm;rate=16000 when sending audio.
  • Decode inline audio data and concatenate int16 PCM chunks before playback.
  • Check turn_complete before playing the full collected response.

Graph View

Backlinks