Generative AI Google Brain

Audio & Speech - Best Practices

8 min read

Audio & Speech — Best Practices

Distilled from 17 notebooks tagged Audio & Speech in the GoogleCloudPlatform/generative-ai repository. The From the notebooks section below cites the per-notebook source for grounding.

Do this

  • Use detailed prompts for generative audio and speech style, including genre, mood, tempo, rhythm, instrumentation, accent, pace, tone, vocal style, and emotional expression.
  • Use negative_prompt with Lyria when specific audio qualities should be excluded.
  • Use seed for deterministic Lyria generation when sample_count is not used.
  • Set response_modalities explicitly, such as AUDIO for TTS or AUDIO and TEXT when both outputs are needed.
  • Use structured Gemini output with response_mime_type set to application/json and a response_schema when downstream synthesis depends on predictable dialogue or script structure.
  • Select documented voices through VoiceConfig, PrebuiltVoiceConfig, MultiSpeakerVoiceConfig, multi_speaker_voice_config, or filtered available voices before assigning speakers or characters.
  • Keep speaker labels and aliases constrained and consistent between the prompt and multi-speaker voice configuration.
  • For streaming synthesis or recognition, send the streaming configuration first, then send text inputs or audio chunks in order.
  • Split long text into sentence chunks before streaming synthesis, and use batch recognition rather than synchronous recognition for longer audio.
  • Use AutoDetectDecodingConfig for transcription so Speech-to-Text can detect audio encoding automatically.
  • Use Cloud Storage URIs for batch recognition inputs and writable Cloud Storage folders for batch outputs.
  • Use language_codes=[“auto”] when language-agnostic transcription is required, and configure SpeakerDiarizationConfig inside RecognitionFeatures for diarization.
  • Decode base64 audio outputs before playback, and handle raw PCM 16-bit 24 kHz audio without WAV headers when using Agent Platform Gemini-TTS outputs.
  • Check for missing audioContent, parse Gemini JSON defensively, handle JSONDecodeError and KeyError, and skip synthesis when no dialogue is generated.
  • For Chirp 3 Instant Custom Voice, separate reference audio from voice talent consent audio, use the required consent script, base64-encode WAV input, and mask stored voice keys in user interfaces.

Avoid this

  • Calling APIs before creating a Google Cloud project, enabling the required API, authenticating in Colab, or setting Application Default Credentials and quota project.
  • Using the wrong endpoint or assuming global availability when Chirp 3, Gemini-TTS, Text-to-Speech, or Speech-to-Text regional availability applies.
  • Setting both seed and sample_count in the same Lyria request.
  • Trying to play returned audio without first decoding base64, wrapping raw PCM, or respecting the expected int16 24 kHz playback format.
  • Assuming multi-speaker behavior is always available: Gemini 3.1 Flash TTS supports up to 2 speakers, Text-to-Speech multi-speaker dialogue may require allowlisting, and different voices per story line require separate TTS calls.
  • Using audio tags too often, placing them away from the phrase they should influence, or using non-English tags when English tags are recommended.
  • Attempting voice cloning without allowlisted access, correct LINEAR16 24000 Hz WAV files, separate consent audio, the required consent script, OAuth access token, or x-goog-user-project header.
  • Assigning more characters than available voices, using replacement PDFs that are not accessible, or writing batch diarization outputs to a non-writable Cloud Storage folder.

From the notebooks

Lyria 2 Music Generation

  • Use detailed prompts describing style, mood, tempo, rhythm, and instrumentation.
  • Use negative_prompt to specify audio qualities to exclude.
  • Use seed for deterministic generation when not using sample_count.
  • Decode bytesBase64Encoded output before playing the audio.
  • Rely on response.raise_for_status() to surface failed API calls.

Lyria 3 Music Generation

  • Set response_modalities to include both AUDIO and TEXT when both outputs are needed.
  • Provide genre, vocal style, and instruments in prompts for text-to-music generation.
  • Use types.Part.from_bytes with the correct image MIME type for image-conditioned generation.
  • Read PROJECT_ID and LOCATION from environment variables when notebook parameters are unset.
  • Clean model text output before displaying Markdown and audio.

Gemini 3.1 Flash Text-to-Speech Generation

  • Use VoiceConfig with PrebuiltVoiceConfig to choose documented voice options.
  • Use response_modalities=[“AUDIO”] when requesting speech output.
  • Use MultiSpeakerVoiceConfig with one SpeakerVoiceConfig per speaker for multi-speaker clips.
  • Place audio tags immediately before the phrase or sentence they should influence.
  • Avoid overusing audio tags and match them to natural changes in tone or pace.

Get started with Chirp 3: Instant custom voice

  • Use application default credentials and set the quota project before API calls.
  • Separate reference audio and voice talent consent audio in the generateVoiceCloningKey request.
  • Base64-encode WAV audio before sending it in the JSON payload.
  • Check for missing audioContent before trying to play generated audio.
  • Encapsulate REST calls and audio conversion in helper functions before wiring the UI.

Get started with Chirp 3 HD voices using Text-to-Speech

  • Use PROJECT_ID from GOOGLE_CLOUD_PROJECT when no explicit project ID is provided.
  • Build API_ENDPOINT from TTS_LOCATION so global and regional endpoints are handled consistently.
  • Use ClientOptions to pass the Text-to-Speech API endpoint explicitly.
  • Split long text into sentence chunks before streaming synthesis.
  • Send the streaming configuration before streaming text inputs.

Get started with Chirp 3 Transcription

  • Use AutoDetectDecodingConfig so the API detects audio encoding.
  • Use a regional Speech-to-Text endpoint based on STT_LOCATION.
  • Use Cloud Storage URIs for batch recognition inputs and outputs.
  • Use language_codes=[“auto”] for language-agnostic transcription.
  • Configure SpeakerDiarizationConfig inside RecognitionFeatures for diarization.

Get started with Gemini-TTS voices using Text-to-Speech

  • Choose Cloud Text-to-Speech API when specific output encodings or bidirectional streaming are needed.
  • Choose Agent Platform API when already using Gemini-TTS from AI Studio or other Agent Platform models.
  • Use natural-language prompts to steer style, accent, pace, tone, and emotional expression.
  • Use multi_speaker_voice_config to assign speaker aliases to specific voices.
  • Stream audio chunks immediately to the frontend in real-time applications.

Create a Multi-Speaker Podcast with Gemini 2.0 & Text-to-Speech

  • Use controlled generation with response_mime_type set to application/json and a response_schema.
  • Parse Gemini output with json.loads and handle JSONDecodeError and KeyError.
  • Skip audio synthesis when no dialogue is generated.
  • Use a regional Text-to-Speech API endpoint based on TTS_LOCATION.
  • Keep speaker labels constrained to R and S for MultiSpeakerMarkup turns.

Narrate a Multi-character Story with Gemini and Text-to-Speech

  • Use response_schema with Gemini to produce structured output for downstream processing.
  • Use a narrator voice for scene settings and title narration.
  • Filter available voices before assigning them to characters.
  • Use one Text-to-Speech call per dialogue line when different voices are required.
  • Combine generated clips with short silence between lines and remove intermediate MP3 files.

Introduction to Long Context Window with Gemini on Vertex AI

  • Start by putting all relevant tokens into the context window when the input fits, while recognizing RAG and summarization can still be relevant.
  • Use count_tokens() before generate_content() for long-context requests.
  • Use Part.from_uri with explicit MIME types for text, video, and audio files.
  • Inspect response.usage_metadata after generation.
  • Use context caching to reduce time and cost for repeated large-context requests.

Getting Started with the Live API Native Audio

  • Use a reusable configuration helper for transcription, proactivity, affective dialog, and system instructions.
  • Use an async context manager for the Live API session lifecycle.
  • Send conversation turns sequentially to maintain session context.
  • Collect input and output transcriptions alongside streamed audio when transcription is enabled.
  • Use system instructions to constrain proactive chime-in behavior and set response persona.

Getting Started with Gemini Live API using Gen AI SDK

  • Use LiveConnectConfig to centralize response modalities, speech configuration, tools, transcription, and realtime input settings.
  • Set end_of_turn to True when generation should start after accumulated client content.
  • Declare tools when initiating the session rather than after the session starts.
  • Use turn_complete to know when to stop collecting an audio response in conversational loops.
  • Send audio_stream_end=True when a realtime audio stream is paused for more than a second.

Gemini Live API Quickstart

  • Use the Google Gen AI SDK for a simplified Live API session and interruption handling.
  • Use WebSockets when you need direct control over the handshake and raw JSON payloads.
  • Stream audio in small chunks and delay briefly to simulate real-time input from a microphone.
  • Set the input MIME type to audio/pcm;rate=16000 when sending audio.
  • Decode inline audio data and concatenate int16 PCM chunks before playback.

Build Your Own AI Podcasting Agent with LangGraph, Gemini, and Chirp 3

  • Defines AgentState with typed workflow fields for task, outline, queries, content, draft, critique, and tool calls.
  • Uses MemorySaver and thread_id to preserve unique workflow execution history.
  • Uses temperature=0 for deterministic agent node outputs.
  • Limits arXiv retrieval with load_max_docs=2 and get_full_documents=False.
  • Prompts the research agent to vary tools and avoid repeating prior sources and queries.

Sheet Music Analysis with Gemini

  • Use environment variables for GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION when notebook parameters are not provided.
  • Set safety settings and a domain-specific system instruction before generation.
  • Use Part.from_uri with explicit MIME types for PDF and audio inputs.
  • Use response_mime_type=“application/json” before parsing model output with json.loads.
  • Copy PDF pages into a writer before adding metadata and saving the file.

Gemini: An Overview of Multimodal Use Cases

  • Pass non-text inputs with explicit MIME types using Part.from_uri.
  • Use context caching for repeated questions over a large codebase instead of resending the same prompt.
  • Set temperature to 0 for factual identification prompts, as shown for the train-line example.
  • Label candidate images in the prompt when asking the model to choose among provided images.
  • Tell the model not to make up information when answers must be grounded only in attached audio or video.

Multimodal Sentiment Analysis with Gemini

  • Set project and location before creating the Gen AI client.
  • Use Part.from_uri with an explicit MIME type for Cloud Storage audio inputs.
  • Compare direct audio analysis with transcript-based analysis to understand modality differences.
  • Use speaker labels in prompts for conversation analysis.
  • Set response_modalities to TEXT when requesting text-only output.

Back to Audio & Speech · Best Practices Map

Graph View

Backlinks