Generative AI Google Brain

gemini-use-cases-video-analysis-scene-transition

2 min read

Identifcation of Scene Transitions in Movies Using Gemini

Source notebook

Repo path: gemini/use-cases/video-analysis/scene_transition.ipynb · Open on GitHub · intermediate

Uses Gemini to detect movie scene transitions from a video and matching VTT subtitles.

Summary

This notebook teaches how to use the Google Gen AI SDK with Vertex AI to analyze a video plus subtitle file for scene boundaries. It defines a Pydantic schema for structured scene outputs, configures Gemini for JSON responses, sends video and VTT Cloud Storage URIs as multimodal inputs, then prints parsed scene transition records.

Key code patterns

Vertex AI GenAI client

from google import genai
client = genai.Client(
    vertexai=True,
    project=PROJECT,
    location="us-central1",
)

Initializes the Google Gen AI SDK against Vertex AI in a specific project and region.

Structured scene schema

class Scene(BaseModel):
    scene_number: int
    start_time: str = Field(..., pattern="^([0-9]{2}):([0-9]{2}):([0-9]{2})$")
    end_time: str = Field(..., pattern="^([0-9]{2}):([0-9]{2}):([0-9]{2})$")
    description: str

Constrains model output to sequential scenes with timestamp fields and descriptions.

JSON response config

config = GenerateContentConfig(
    temperature=0,
    top_p=1,
    max_output_tokens=8000,
    response_mime_type="application/json",
    response_schema=list[Scene],
)

Requests deterministic, schema-constrained JSON suitable for parsing.

Multimodal video and VTT prompt

response = client.models.generate_content(
    model=MODEL_ID,
    contents=[
        Part.from_text(text=base_instructions),
        "Apply the scene transition instructions above.",
        Part.from_uri(file_uri=input_video_path, mime_type="video/mp4"),
        Part.from_uri(file_uri=input_vtt_path, mime_type="text/vtt"),
    ],
    config=config,
)

Combines task instructions, video content, and subtitle timing in one Gemini request.

Models & APIs used

  • Models: gemini-2.0-flash
  • APIs / services: Vertex AI, Cloud Storage
  • SDKs / libraries: google-genai, pydantic

When to use this

Use this pattern when you need structured scene segmentation from a video using both visual content and subtitles.

Gotchas & caveats

  • Requires a Google Cloud project with the Vertex AI API enabled.
  • The notebook uses location us-central1.
  • Cloud Storage input URIs must reference accessible video and VTT files.
  • Timestamp outputs are constrained to HH:MM:SS by the Pydantic schema.
  • The prompt instructs that scene timestamps must fall within the input video’s boundaries.

Best practices

  • Use temperature 0 for consistent scene boundary extraction.
  • Provide both video and VTT inputs to combine visual and dialogue cues.
  • Use response_mime_type application/json with response_schema for structured outputs.
  • Define explicit criteria for narrative, visual, dialogue, audio, and cohesion signals.
  • Do not count jump-cuts or insert shots as transitions unless they indicate meaningful narrative shifts.

Graph View

Backlinks