Generative AI Google Brain

gemini-use-cases-multimodal-data-curation-splitting-and-transcoding

4 min read

Data Curation Pipeline: Splitting and Transcoding

Source notebook

Repo path: gemini/use-cases/multimodal-data-curation/splitting_and_transcoding.ipynb · Open on GitHub · intermediate

Builds a video curation pipeline for metadata filtering, splitting, scene detection, and transcoding.

Summary

The notebook teaches foundational video data curation steps for VLM pre-training using a subset of VidGen-1M videos stored in Cloud Storage. It lists and filters MP4 files by metadata from ffprobe, splits videos into fixed-length or scene-based segments, and transcodes segments with FFmpeg. It also shows an alternative asynchronous workflow using the Google Cloud Transcoder API with edit lists, streams, and mux streams.

Key code patterns

Initialize clients

from google import genai
from google.cloud import storage
 
LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "us-central1")
storage_client = storage.Client()
client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)

Sets project, region, Cloud Storage access, and a Vertex AI GenAI client.

List videos in GCS

def list_videos_in_bucket(bucket_name, prefix):
    blobs = storage_client.bucket(bucket_name).list_blobs(prefix=prefix)
    return [blob.name for blob in blobs if blob.name.lower().endswith(".mp4")]

Finds MP4 source files in the VidGen-1M Cloud Storage bucket.

Extract metadata with ffprobe

command = ["ffprobe", "-i", "pipe:0", "-show_format", "-show_streams", "-print_format", "json"]
process = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output, error = process.communicate(input=video_stream.getvalue())
metadata = json.loads(output.decode("utf-8"))

Downloads video bytes from Cloud Storage and parses duration, format, and stream metadata.

Filter videos

if duration is not None and duration >= min_duration:
    for stream in streams:
        if stream.get("codec_type") == "video":
            height = stream.get("height")
            if height is not None and height >= min_resolution_height:
                filtered_videos.append(video_file)

Keeps videos that meet minimum duration and resolution criteria.

Fixed-length splitting

ffmpeg_command = [
    "ffmpeg", "-i", downloaded_file_path, "-map", "0", "-c", "copy",
    "-f", "segment", "-segment_time", str(segment_duration_seconds),
    "-reset_timestamps", "1", output_pattern,
]

Uses FFmpeg segment muxing and stream copy for robust fixed-duration clips.

Scene detection

video_object = open_video(video_source_path)
scene_manager = SceneManager(stats_manager=None)
scene_manager.add_detector(ContentDetector(threshold=20))
scene_manager.detect_scenes(video_object)
scene_list = scene_manager.get_scene_list()

Finds content-aware scene boundaries using PySceneDetect.

Transcode segments

ffmpeg_command = [
    "ffmpeg", "-i", input_segment_path,
    "-c:v", "libx264", "-b:v", "1500k",
    "-c:a", "aac", "-b:a", "128k",
    "-vf", "scale=1280x720", "-r", "25", output_path,
]

Standardizes segment codec, bitrate, resolution, frame rate, and container.

Submit Transcoder job

parent = f"projects/{PROJECT_ID}/locations/{LOCATION}"
job = {"input_uri": gcs_input_uri, "output_uri": gcs_output_uri, "config": config}
response = transcoder_client.create_job(parent=parent, job=job)

Shows managed asynchronous splitting and transcoding with Cloud Transcoder API.

Models & APIs used

  • APIs / services: Vertex AI, Cloud Storage, Transcoder API
  • SDKs / libraries: google-genai, google-cloud-storage, google-cloud-transcoder, PySceneDetect

When to use this

Use this pattern when preparing raw video datasets into smaller standardized clips for VLM training, filtering, captioning, or analysis.

Gotchas & caveats

  • Requires an existing Google Cloud project with the Vertex AI API enabled.
  • Colab users must authenticate with google.colab.auth.authenticate_user().
  • PROJECT_ID defaults to GOOGLE_CLOUD_PROJECT and LOCATION defaults to GOOGLE_CLOUD_REGION or us-central1.
  • Cloud Storage blobs must exist and be accessible with current credentials.
  • ffprobe and ffmpeg must be installed and available on PATH.
  • ffprobe may report partial file errors or produce JSON decode failures.
  • Scene detection threshold requires tuning for each video’s visual characteristics.
  • MoviePy was noted as less reliable for short fixed-length segments in this notebook.
  • Google Cloud Transcoder API jobs are asynchronous and require checking the GCS output path.

Best practices

  • Filter videos by duration and resolution before downstream processing.
  • Use FFmpeg directly via subprocess for robust fixed-length splitting with stream copy.
  • Clean up temporary files and directories after local video processing.
  • Sort segment files before batch transcoding for consistent processing order.
  • Test transcoding parameters on a sample of video data before applying them broadly.
  • Use MP4, H.264, AAC, and practical bitrate or resolution settings for broad compatibility.
  • Use scene detection when semantically meaningful video segments are needed.

Graph View

Backlinks