Generative AI Google Brain

gemini-use-cases-multimodal-data-curation-quality-filtering

3 min read

Video Data Curation - Video Quality Filtering

Source notebook

Repo path: gemini/use-cases/multimodal-data-curation/quality-filtering.ipynb · Open on GitHub · intermediate

Filters video datasets by metadata, OCR/watermarks, aesthetics, and motion scores for curation.

Summary

This notebook teaches a video data curation workflow for removing low-quality clips before model training. It loads video blobs from Cloud Storage, extracts metadata with PyAV, detects visible text and watermarks with Gemini structured JSON output, scores aesthetics with CLIP plus an MLP, and estimates motion using sparse optical flow in OpenCV.

Key code patterns

Initialize clients

storage_client = storage.Client(project=PROJECT_ID)
gemini_client = genai.Client(
    vertexai=True,
    project=PROJECT_ID,
    location=REGION,
)

Connects the notebook to Cloud Storage and Vertex AI through the Google GenAI SDK.

List video blobs

def load_video_paths(bucket: str, num_videos: int) -> list[str]:
    video_blobs = []
    for i, blob in enumerate(storage_client.list_blobs(bucket)):
        if i >= num_videos:
            break
        video_blobs.append(blob.name)
    return video_blobs

Reads a small sample of video object names from a Cloud Storage bucket for inspection.

Extract metadata with PyAV

container = av.open(video_file_object)
video_stream = container.streams.video[0]
height = video_stream.height
width = video_stream.width
avg_fps = float(video_stream.average_rate)
common_divisor = math.gcd(width, height)
aspect_ratio = f"{width // common_divisor}:{height // common_divisor}"

Derives core video quality fields such as resolution, FPS, and aspect ratio from the video stream.

Gemini structured video OCR

response = gemini_client.models.generate_content(
    model="gemini-2.5-flash",
    contents=[
        types.Part.from_text(text="Watch this video:"),
        types.Part.from_uri(file_uri=video_uri, mime_type="video/mp4"),
    ],
    config=types.GenerateContentConfig(
        temperature=0.0,
        response_mime_type="application/json",
    ),
)

Uses Gemini on a Cloud Storage video URI and asks for machine-readable text and watermark fields.

Estimate sparse motion

p0 = cv.goodFeaturesToTrack(old_gray, mask=None, **feature_params)
p1, st, err = cv.calcOpticalFlowPyrLK(
    old_gray, frame_gray, p0, None, **lk_params
)
good_new = p1[st == 1]
good_old = p0[st == 1]
distances = np.linalg.norm(good_new - good_old, axis=1)

Computes average frame-to-frame feature displacement as a coarse motion score.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI, Cloud Storage
  • SDKs / libraries: google-genai, google-cloud-storage, pyav, opencv-python, numpy, torch, transformers, pillow

When to use this

Use this pattern when curating video datasets and you need practical filters before spending compute on training or downstream modeling.

Gotchas & caveats

  • Requires a Google Cloud project ID, Vertex AI region, and Cloud Storage bucket name.
  • Colab users must authenticate and restart the runtime after package installation.
  • The notebook assumes videos are already stored in a Cloud Storage bucket.
  • The aesthetic scoring section requires downloading sac+logos+ava1-l14-linearMSE.pth before running.
  • Quality thresholds are not universal and should be chosen by manually inspecting low-scoring samples.

Best practices

  • Discard low-quality clips to use limited modeling compute efficiently.
  • Use metadata filters such as FPS, duration, resolution, brightness, and aspect ratio for downstream modeling requirements.
  • Use structured JSON output for separating detected text from watermark descriptions.
  • Set Gemini temperature to 0.0 for deterministic extraction-style calls.
  • Manually inspect low-scoring videos to choose an aesthetic score threshold.

Graph View

Backlinks