Generative AI Google Brain

gemini-multimodal-live-api-intro-multimodal-live-api-genai-sdk

3 min read

Getting Started with Gemini Live API using Gen AI SDK

Source notebook

Repo path: gemini/multimodal-live-api/intro_multimodal_live_api_genai_sdk.ipynb · Open on GitHub · intermediate

Introduces Gemini Live API audio sessions, tools, transcription, and VAD with the Google Gen AI SDK.

Summary

This notebook teaches how to use the Gemini Live API in Vertex AI through the Google Gen AI SDK. It walks through project setup, creating an enterprise GenAI client, opening async live sessions, sending text or realtime audio, receiving audio responses, and enabling tools such as function calling, code execution, and Google Search. It also demonstrates input/output audio transcription and configuring automatic voice activity detection.

Key code patterns

Create Vertex AI GenAI client

PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")
LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "us-central1")
client = genai.Client(
    enterprise=True,
    project=PROJECT_ID,
    location=LOCATION,
)

Initializes the Google Gen AI SDK client for Vertex AI using project and region settings.

Open live audio session

config = LiveConnectConfig(response_modalities=["audio"])
async with client.aio.live.connect(
    model=MODEL_ID,
    config=config,
) as session:
    await session.send_client_content(
        turns=Content(role="user", parts=[Part(text=text_input)])
    )

Shows the core async WebSocket session pattern for sending user turns to the Live API.

Collect audio response parts

audio_data = []
async for message in session.receive():
    if message.server_content.model_turn:
        for part in message.server_content.model_turn.parts:
            if part.inline_data:
                audio_data.append(np.frombuffer(part.inline_data.data, dtype=np.int16))
if audio_data:
    display(Audio(np.concatenate(audio_data), rate=24000, autoplay=True))

Streams model audio chunks, converts them to int16 arrays, and plays concatenated audio.

Register a Python function tool

def get_current_weather(location: str) -> str:
    weather_map = {"Boston, MA": "snowing"}
    return weather_map.get(location, "unknown")
 
config = LiveConnectConfig(
    response_modalities=["audio"],
    tools=[get_current_weather],
)

Demonstrates declaring a callable tool at session start so the model can request function calls.

Configure VAD realtime input

config = LiveConnectConfig(
    response_modalities=["audio"],
    realtime_input_config=RealtimeInputConfig(
        automatic_activity_detection=AutomaticActivityDetection(
            disabled=False,
            silence_duration_ms=100,
        )
    ),
)

Configures automatic activity detection for realtime audio and interruption-aware conversations.

Models & APIs used

When to use this

Use this pattern to build low-latency Gemini voice or audio applications with live sessions, tools, transcription, and VAD.

Gotchas & caveats

  • Vertex AI API must be enabled for the Google Cloud project.
  • Colab requires explicit auth.authenticate_user() authentication.
  • PROJECT_ID falls back to GOOGLE_CLOUD_PROJECT and LOCATION defaults to us-central1.
  • Live API sessions are single WebSocket connections, and context is erased when the session terminates.
  • All function tools must be declared at session start.
  • The notebook states that currently only one tool is supported in the API.
  • Native audio models automatically choose language and do not support explicitly setting the language code.
  • The VAD example uses Blob without importing it in the shown import cell.

Best practices

  • Use LiveConnectConfig to centralize response modalities, speech configuration, tools, transcription, and realtime input settings.
  • Set end_of_turn to True when generation should start after accumulated client content.
  • Declare tools when initiating the session rather than after the session starts.
  • Use turn_complete to know when to stop collecting an audio response in conversational loops.
  • Send audio_stream_end=True when a realtime audio stream is paused for more than a second.

Graph View

Backlinks