Generative AI Google Brain

audio-speech-use-cases-storytelling-storytelling

3 min read

Narrate a Multi-character Story with Gemini and Text-to-Speech

Source notebook

Repo path: audio/speech/use-cases/storytelling/storytelling.ipynb · Open on GitHub · intermediate

Generates a Gemini play script and narrates it with distinct Text-to-Speech voices per character.

Summary

This notebook teaches how to create a structured multi-character play with Gemini and turn it into an audio performance. It generates or loads a play, lists Chirp3 Text-to-Speech voices, maps each character to a voice, synthesizes each line as MP3, and combines the clips into one final audio file.

Key code patterns

Create Gemini client

from google import genai
 
PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")
LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "global")
client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)

Initializes the Gemini API on Agent Platform with project and location settings.

Structured story schema

class DialogueLine(BaseModel):
    speaker: str
    line: str
 
class Scene(BaseModel):
    setting: str
    dialogue: list[DialogueLine]
 
class Story(BaseModel):
    title: str
    characters: list[Character]
    scenes: list[Scene]

Defines the JSON structure Gemini must return for downstream voice assignment and audio synthesis.

Generate parsed story

response = client.models.generate_content(
    model=MODEL_ID,
    contents=PROMPT,
    config=GenerateContentConfig(
        system_instruction=SYSTEM_INSTRUCTION,
        response_mime_type="application/json",
        response_schema=Story,
    ),
)
story = response.parsed

Uses Gemini with a response schema so the generated play can be consumed as typed data.

Configure regional TTS client

api_endpoint = "texttospeech.googleapis.com"
if TTS_LOCATION != "global":
    api_endpoint = f"{TTS_LOCATION}-{api_endpoint}"
 
tts_client = texttospeech.TextToSpeechClient(
    client_options=ClientOptions(api_endpoint=api_endpoint)
)

Builds a Text-to-Speech client that can target global or regional endpoints.

Synthesize one line per voice

response = tts_client.synthesize_speech(
    input=texttospeech.SynthesisInput(text=text),
    voice=texttospeech.VoiceSelectionParams(language_code=language_code, name=voice_name),
    audio_config=texttospeech.AudioConfig(audio_encoding=texttospeech.AudioEncoding.MP3),
)
with open(output_file, "wb") as f:
    f.write(response.audio_content)

Handles the notebook’s constraint that each Text-to-Speech call uses one voice.

Models & APIs used

When to use this

Use this pattern when converting a generated multi-speaker script into narrated audio with separate voices per character.

Gotchas & caveats

  • Requires Google Cloud authentication and setting the active gcloud project.
  • Uses billable Gemini API in Vertex AI, Text-to-Speech, and Cloud Storage components.
  • Text-to-Speech available regions must be checked before setting TTS_LOCATION.
  • Text-to-Speech can only create audio with one voice per API call, so each line is synthesized separately.
  • Mac users need FFmpeg installed for pydub MP3 handling.
  • Voice assignment can fail if there are more characters than available voices.

Best practices

  • Use response_schema with Gemini to produce structured output for downstream processing.
  • Use a narrator voice for scene settings and title narration.
  • Filter available voices before assigning them to characters.
  • Use one Text-to-Speech call per dialogue line when different voices are required.
  • Combine generated clips with short silence between lines and remove intermediate MP3 files.

Graph View

Backlinks