Generative AI Google Brain

gemini-use-cases-multimodal-data-curation-captioning

3 min read

Video Captioning with Gemini

Source notebook

Repo path: gemini/use-cases/multimodal-data-curation/captioning.ipynb · Open on GitHub · intermediate

Uses Gemini on Vertex AI to generate rich multimodal captions from a video in Cloud Storage.

Summary

This notebook teaches how to caption video with Gemini using the Google Gen AI SDK on Vertex AI. It configures project, location, and a Gemini client, defines detailed system instructions, sends text plus a video URI to Gemini, optionally counts video tokens, and returns caption text with usage metadata. The example emphasizes visual, on-screen text, and audio-derived details for multimodal data curation.

Key code patterns

Vertex AI Gen AI client

from google import genai
 
client = genai.Client(
    vertexai=True,
    project=PROJECT_ID,
    location=LOCATION,
)

Initializes Google Gen AI SDK access through Vertex AI with project and region.

Video plus text contents

contents = [
    types.Part.from_text(text="Caption this video"),
    types.Part.from_uri(
        file_uri=video_uri,
        mime_type="video/mp4",
    ),
]

Combines a text task instruction with a Cloud Storage video URI as multimodal input.

Video token counting

video_token_count = client.models.count_tokens(
    model="gemini-2.0-flash-001",
    contents=[types.Part.from_uri(
        file_uri=video_uri,
        mime_type="video/mp4",
    )],
).total_tokens

Measures video-only token usage before generation when requested.

Caption generation config

response = client.models.generate_content(
    model=model,
    contents=contents,
    config=types.GenerateContentConfig(
        system_instruction=system_instructions,
        temperature=temperature,
    ),
)

Uses system instructions and temperature to guide detailed video caption output.

Return text and usage metadata

return {
    "prompt_token_count": response.usage_metadata.prompt_token_count,
    "candidates_token_count": response.usage_metadata.candidates_token_count,
    "total_token_count": response.usage_metadata.total_token_count,
    "model_version": response.model_version,
    "text": response.candidates[0].content.parts[0].text,
}

Captures generated caption text together with token counts and model version.

Models & APIs used

When to use this

Use this pattern when you need detailed multimodal video captions from Cloud Storage videos using Gemini on Vertex AI.

Gotchas & caveats

  • Colab users must authenticate with google.colab.auth.authenticate_user().
  • A Google Cloud project is required and the Vertex AI API must be enabled.
  • PROJECT_ID falls back to GOOGLE_CLOUD_PROJECT if the placeholder is not replaced.
  • LOCATION falls back to GOOGLE_CLOUD_REGION or us-central1.
  • The video input is a gs:// URI with mime_type video/mp4, so the object must be accessible to the caller.
  • The notebook warns that using Gemini output to train competing models may violate Gemini API Additional Terms of Service.

Best practices

  • Use a detailed system prompt that specifies perspective, camera movement, subject motion, setting, lighting, body language, and visible text.
  • Return only the caption by instructing the model not to send a preamble.
  • Count video tokens separately when token accounting matters.
  • Return usage metadata, model version, generated text, and video-only token count for traceability.
  • Review the sample video before captioning to understand visual, textual, and audio elements.

Graph View

Backlinks