Generative AI Google Brain

vision-getting-started-veo3-advanced-controls

3 min read

Veo 3.1 Advanced Controls

Source notebook

Repo path: vision/getting-started/veo3_advanced_controls.ipynb · Open on GitHub · intermediate

Shows Veo 3.1 frame interpolation and video extension with Google Gen AI SDK on Agent Platform.

Summary

This notebook teaches how to call Veo 3.1 video generation models from a notebook using the Google Gen AI SDK for Python. It demonstrates frame interpolation from local image files and Cloud Storage image URIs, polling long-running video operations, saving outputs as bytes or to Cloud Storage, and extending an existing Cloud Storage video.

Key code patterns

Create enterprise GenAI client

from google import genai
 
PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")
LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "us-central1")
client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)

Configures the SDK to call Agent Platform with a Google Cloud project and region.

Frame interpolation from files

operation = client.models.generate_videos(
    model="veo-3.1-generate-001",
    prompt=prompt,
    image=types.Image.from_file(location=first_frame),
    config=types.GenerateVideosConfig(
        last_frame=types.Image.from_file(location=last_frame),
        aspect_ratio="9:16",
        resolution="1080p",
        duration_seconds=8,
        generate_audio=True,
    ),
)

Uses first and last frames so Veo generates the in-between video content.

Poll video operation

while not operation.done:
    time.sleep(10)
    operation = client.operations.get(operation)
 
if operation.response:
    show_video(operation.result.generated_videos[0].video.video_bytes)

Video generation is asynchronous, so callers must poll before reading generated video output.

Use Cloud Storage inputs and output

operation = client.models.generate_videos(
    model=video_model,
    prompt=prompt,
    image=types.Image(gcs_uri=first_frame_gcs, mime_type="image/png"),
    config=types.GenerateVideosConfig(
        last_frame=types.Image(gcs_uri=last_frame_gcs, mime_type="image/png"),
        output_gcs_uri=output_gcs,
    ),
)

Avoids local downloads and writes larger generated videos directly to Cloud Storage.

Extend an existing video

operation = client.models.generate_videos(
    model="veo-3.1-generate-preview",
    prompt=prompt,
    video=types.Video(uri=video_gcs, mime_type="video/mp4"),
    config=types.GenerateVideosConfig(
        output_gcs_uri=output_gcs,
        duration_seconds=7,
        generate_audio=True,
    ),
)

Uses a video input URI to generate an additional segment from the existing clip.

Models & APIs used

When to use this

Use this pattern when building notebook or prototype workflows for Veo 3.1 video generation with frame controls, audio, Cloud Storage IO, or video extension.

Gotchas & caveats

  • Requires an existing Google Cloud project and the Agent Platform API enabled.
  • Colab requires auth.authenticate_user().
  • PROJECT_ID must be set directly or via GOOGLE_CLOUD_PROJECT.
  • LOCATION defaults to us-central1 when GOOGLE_CLOUD_REGION is unset.
  • Cloud Storage output requires replacing gs://[your-bucket-path] with a valid bucket path.
  • 4k generation can introduce latency up to several minutes.
  • Frame interpolation works best when starting and ending frames are similar in nature.
  • Video extension example saves output to Cloud Storage because outputs are bigger.

Best practices

  • Use Cloud Storage URIs directly for remote images instead of downloading when appropriate.
  • Poll long-running operations with client.operations.get before reading results.
  • Set output_gcs_uri for larger generated videos and video extension outputs.
  • Specify mime_type when passing Cloud Storage image or video URIs.
  • Use SynthID-watermarked Veo outputs as noted by the notebook.

Graph View

Backlinks