Generative AI Google Brain

gemini-use-cases-video-analysis-multimodal-video-transcription

3 min read

Unlocking Multimodal Video Transcription with Gemini

Source notebook

Repo path: gemini/use-cases/video-analysis/multimodal_video_transcription.ipynb · Open on GitHub · intermediate

Uses Gemini to build multimodal video transcripts with voice IDs, speaker names, and structured JSON output.

Summary

This notebook teaches a prompt-based workflow for transcribing video with Gemini using audio, visual, and time cues in one request. It starts from simple transcription prompts, shows why direct speaker-labeled transcription fails on more complex videos, then introduces tabular extraction to decouple transcripts from speaker identification. It finalizes the workflow with Google GenAI SDK structured output using Pydantic schemas and deterministic generation settings.

Key code patterns

Create GenAI client

from google import genai
 
check_environment()
client = genai.Client()
check_configuration(client)

Uses environment-based configuration for either Agent Platform or Google AI Studio.

Deterministic config

DEFAULT_CONFIG = GenerateContentConfig(
    temperature=0.0,
    top_p=0.0,
    seed=42,
    thinking_config=ThinkingConfig(
        thinking_level=ThinkingLevel.MINIMAL
    ),
)

Sets low-randomness parameters for factual data extraction from video.

Video content request

contents = [video_part, prompt]
response = client.models.generate_content(
    model=model_id,
    contents=contents,
    config=config,
)

Sends video plus prompt to Gemini in a single multimodal request.

Structured output schema

class Transcript(pydantic.BaseModel):
    start: str
    text: str
    voice: int
 
class Speaker(pydantic.BaseModel):
    voice: int
    name: str

Moves JSON field definitions from the prompt into typed Pydantic models.

Parsed response

if isinstance(response.parsed, VideoTranscription):
    video_transcription = response.parsed
else:
    video_transcription = VideoTranscription()

Uses SDK parsing to avoid manual JSON extraction from plain text responses.

Models & APIs used

  • Models: gemini-3.1-flash-lite
  • APIs / services: Gemini API, Agent Platform, Google AI Studio, Cloud Storage
  • SDKs / libraries: google-genai, pandas, pydantic, tenacity

When to use this

Use this pattern when a video transcription task needs speech text, timecodes, voice separation, and speaker names from audio and visual cues.

Gotchas & caveats

  • Agent Platform requires a Google Cloud project and the Agent Platform API enabled.
  • Google AI Studio requires a Gemini API key.
  • Preview models require location global; generally available models can use supported Google model endpoint locations.
  • YouTube captions, subtitles, transcripts, and metadata are not provided to Gemini for YouTube video analysis.
  • Direct speaker-labeled transcription can fail on longer or more complex videos.
  • Prompt output specs should not contradict the structured response schema.

Best practices

  • Use environment variables or Colab Secrets instead of hardcoding API configuration.
  • Start with simple prompts to observe Gemini’s natural behavior before refining instructions.
  • Craft prompts iteratively, precisely, and concisely.
  • Decouple transcripts and speakers into linked tables with a consistent voice ID.
  • Start transcript generation with audio-focused work before extracting speaker names from visual and audio cues.
  • Use response_mime_type application/json and response_schema for consistent structured outputs.
  • Use Pydantic classes to keep schema maintenance separate from prompt logic.

Graph View

Backlinks