Generative AI Google Brain

gemini-multimodal-live-api-intro-live-api-native-audio

3 min read

Getting Started with the Live API Native Audio

Source notebook

Repo path: gemini/multimodal-live-api/intro_live_api_native_audio.ipynb · Open on GitHub · intro

Connects to Gemini Live API native audio with proactive chime-in and affective dialog examples.

Summary

This notebook teaches how to use the Google Gen AI SDK for Python to connect to the Gemini Live API through Vertex AI. It builds reusable helpers to configure native audio sessions, send text turns, receive streamed audio and transcriptions, and play 24 kHz audio responses. It demonstrates Proactive Audio with an Italian cooking system instruction and Affective Dialog with a senior technical advisor persona.

Key code patterns

Create GenAI client

from google import genai
 
PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")
LOCATION = "us-central1"
client = genai.Client(
    enterprise=True,
    project=PROJECT_ID,
    location=LOCATION,
)

Creates the Vertex AI-backed Google Gen AI SDK client used by Live API sessions.

Configure native audio session

config = LiveConnectConfig(
    response_modalities=["AUDIO"],
    system_instruction=system_instruction,
    input_audio_transcription=AudioTranscriptionConfig(),
    output_audio_transcription=AudioTranscriptionConfig(),
    proactivity=ProactivityConfig(proactive_audio=True),
    enable_affective_dialog=enable_affective_dialog,
)

Centralizes audio response mode, transcription, proactive audio, and affective dialog options.

Send a user turn

await session.send_client_content(
    turns=Content(
        role="user",
        parts=[Part(text=text_input)],
    )
)

Shows the notebook’s text-to-live-session turn format using Content and Part.

Receive streamed audio

async for message in session.receive():
    if message.server_content.output_transcription:
        output_transcriptions.append(
            message.server_content.output_transcription.text
        )
    if message.server_content.model_turn:
        for part in message.server_content.model_turn.parts:
            if part.inline_data:
                audio_data.append(np.frombuffer(part.inline_data.data, dtype=np.int16))

Processes streaming server messages into transcriptions and audio chunks.

Manage Live API session

async with client.aio.live.connect(
    model=model_id,
    config=config,
) as session:
    all_results = []
    for turn in turns:
        result = await send_and_receive_turn(session, turn)
        all_results.append(result)

Uses an async context manager to open, use, and close the Live API session.

Models & APIs used

When to use this

Use this pattern when building Gemini Live API audio dialogs that stream model speech and need proactive chime-in or affective dialog behavior.

Gotchas & caveats

  • Requires an existing Google Cloud project with the Vertex AI API enabled.
  • In Colab, authenticate with google.colab.auth.authenticate_user().
  • PROJECT_ID must be provided or available as GOOGLE_CLOUD_PROJECT.
  • The notebook uses LOCATION = “us-central1” for the client.
  • Proactive Audio requires ProactivityConfig(proactive_audio=True).
  • Audio chunks are assumed to be np.int16 and played at the required 24000 Hz rate.
  • With proactive audio and system instructions, some turns may intentionally return no audio.

Best practices

  • Use a reusable configuration helper for transcription, proactivity, affective dialog, and system instructions.
  • Use an async context manager for the Live API session lifecycle.
  • Send conversation turns sequentially to maintain session context.
  • Collect input and output transcriptions alongside streamed audio when transcription is enabled.
  • Use system instructions to constrain proactive chime-in behavior and set response persona.

Graph View

Backlinks