Generative AI Google Brain

Audio & Speech

5 min read

Audio & Speech

Audio & Speech on Google Cloud covers generative music, text-to-speech, custom voices, transcription, and multi-speaker narration workflows using Google Cloud generative AI models and speech APIs. The notebooks center on Lyria, Gemini TTS, Chirp 3, Cloud Text-to-Speech, Speech-to-Text V2, Vertex AI, and the Agent Platform API.

Synthesized across 17 notebooks. See the source pages under Notebooks below for line-level grounding.

This area spans two main production patterns: generating audio from prompts or structured text, and extracting text or speaker information from audio. Lyria is used for music generation, Gemini and Chirp 3 are used for speech synthesis, and Chirp 3 with Speech-to-Text V2 is used for transcription.

Most notebooks share the same operational foundation: create or select a Google Cloud project, enable the required API, authenticate in Colab or with Application Default Credentials, set a quota project, and choose a supported global or regional endpoint. Regional availability, allowlisting, billing status, and endpoint choice are recurring constraints.

Audio handling is a first-class engineering concern. Outputs may arrive as base64-encoded WAV, MP3, int16 chunks, or raw PCM 16-bit 24 kHz audio without WAV headers, so playback and frontend delivery usually require explicit decoding, conversion, chunking, or wrapping.

Canonical workflow

  1. Create or select a Google Cloud project, enable the required API, authenticate, and configure Application Default Credentials and quota project when needed.
  2. Choose the model and API path for the task: Lyria for music, Gemini or Chirp 3 with Cloud Text-to-Speech for synthesis, or Chirp 3 with Speech-to-Text V2 for transcription.
  3. Set PROJECT_ID and LOCATION or service-specific locations, then build the correct global or regional endpoint for Text-to-Speech or Speech-to-Text.
  4. Prepare inputs such as detailed prompts, text, PDFs, audio files, image parts, Cloud Storage URIs, consent audio, or structured Gemini response schemas.
  5. Configure request details including response modalities, voice configuration, speaker aliases, language detection, diarization, output encoding, seed, negative prompts, and streaming configuration.
  6. Execute the request using the correct mode: synchronous, batch, single request with multiple responses, bidirectional streaming, or streaming recognition with an initial config followed by chunks.
  7. Decode, convert, validate, play, stream, or store the audio or transcript output, handling missing audio content, JSON parsing errors, and service errors explicitly.
  8. For composed outputs such as podcasts or stories, synthesize by turn or line as required, merge clips with silence, and clean up intermediate audio files.

Best practices

See Audio & Speech - Best Practices for the full distilled guide.

  • Use detailed prompts for generative audio and speech style, including genre, mood, tempo, rhythm, instrumentation, accent, pace, tone, vocal style, and emotional expression.
  • Use negative_prompt with Lyria when specific audio qualities should be excluded.
  • Use seed for deterministic Lyria generation when sample_count is not used.
  • Set response_modalities explicitly, such as AUDIO for TTS or AUDIO and TEXT when both outputs are needed.
  • Use structured Gemini output with response_mime_type set to application/json and a response_schema when downstream synthesis depends on predictable dialogue or script structure.
  • Select documented voices through VoiceConfig, PrebuiltVoiceConfig, MultiSpeakerVoiceConfig, multi_speaker_voice_config, or filtered available voices before assigning speakers or characters.

Key models & APIs

  • lyria-002, lyria-3-clip-preview, lyria-3-pro-preview, gemini-3.1-flash-tts-preview, gemini-2.5-flash-tts, gemini-3.5-flash, gemini-2.0-flash-001, Chirp 3 Instant Custom Voice, Chirp 3 HD voices, including en-US-Chirp3-HD-Aoede, en-GB-Chirp3-HD-Zubenelgenubi, and en-GB-Chirp3-HD-Umbriel, chirp_3, Cloud Text-to-Speech API, Speech-to-Text V2 API, Vertex AI API, Agent Platform API through aiplatform.googleapis.com, Google Gen AI SDK, google-cloud-texttospeech

Notebooks (17)

Audio (9)

Gemini (8)

Part of the Generative AI Google Brain · Audio & Speech - Best Practices

Graph View

Backlinks