Generative AI Google Brain

audio-speech-use-cases-podcast-multi-speaker-podcast

3 min read

Create a Multi-Speaker Podcast with Gemini 2.0 & Text-to-Speech

Source notebook

Repo path: audio/speech/use-cases/podcast/multi-speaker-podcast.ipynb · Open on GitHub · intermediate

Generates a two-speaker podcast from a PDF using Gemini JSON output and Text-to-Speech MP3 synthesis.

Summary

This notebook teaches how to use Gemini API in Vertex AI to turn PDF content into a structured two-speaker podcast script. It loads a PDF URI, prompts Gemini 2.0 Flash with controlled JSON generation, parses the dialogue array, then uses Text-to-Speech MultiSpeakerMarkup with a studio multi-speaker voice to synthesize an MP3. The workflow includes Colab authentication, Google Cloud project setup, regional Text-to-Speech endpoint selection, and playback in the notebook.

Key code patterns

Create Vertex AI Gemini client

from google import genai
 
client = genai.Client(
    enterprise=True,
    project=PROJECT_ID,
    location=VERTEXAI_LOCATION,
)

Connects the Google GenAI SDK to Vertex AI using the selected project and region.

Generate controlled JSON dialogue

response = client.models.generate_content(
    model=MODEL_ID,
    contents=[prompt, Part.from_uri(file_uri=file_uri, mime_type="application/pdf")],
    config=GenerateContentConfig(
        response_mime_type="application/json",
        response_schema=response_schema,
    ),
)

Uses a PDF input and response schema so Gemini returns parseable dialogue data.

Parse dialogue safely

try:
    generated_json = json.loads(response.text)
    return generated_json["dialogue"]
except (json.JSONDecodeError, KeyError):
    return []

Handles malformed or missing JSON fields before calling speech synthesis.

Synthesize multi-speaker audio

multi_speaker_markup = texttospeech.MultiSpeakerMarkup(
    turns=[texttospeech.MultiSpeakerMarkup.Turn(
        text=turn["line"], speaker=turn["speaker"]
    ) for turn in dialogue]
)

Maps Gemini dialogue turns into Text-to-Speech multi-speaker markup.

Use regional Text-to-Speech endpoint

tts_client = texttospeech.TextToSpeechClient(
    client_options=ClientOptions(
        api_endpoint=f"{TTS_LOCATION}-texttospeech.googleapis.com"
    )
)

Configures Text-to-Speech for the selected location such as us.

Models & APIs used

  • Models: gemini-2.0-flash-001
  • APIs / services: Vertex AI, Text-to-Speech API, Cloud Storage
  • SDKs / libraries: google-genai, google-cloud-texttospeech

When to use this

Use this pattern to convert PDF content into a short multi-speaker audio podcast with structured Gemini output and Text-to-Speech synthesis.

Gotchas & caveats

  • Multi-speaker dialogue generation in Text-to-Speech is only available to allowlisted projects.
  • Vertex AI API must be enabled for the Google Cloud project.
  • Colab requires explicit Google Cloud authentication.
  • Application default credentials and quota project are configured with gcloud commands.
  • Text-to-Speech availability depends on supported regions and endpoints.
  • The sample PDF URI is a Cloud Storage path, and replacement PDFs must be publicly accessible if using a URL.
  • The notebook restarts the runtime after installing packages.

Best practices

  • Use controlled generation with response_mime_type set to application/json and a response_schema.
  • Parse Gemini output with json.loads and handle JSONDecodeError and KeyError.
  • Skip audio synthesis when no dialogue is generated.
  • Use a regional Text-to-Speech API endpoint based on TTS_LOCATION.
  • Keep speaker labels constrained to R and S for MultiSpeakerMarkup turns.

Graph View

Backlinks