Generative AI Google Brain

gemini-use-cases-multimodal-data-curation-semantic-deduplication

3 min read

Data Curation Pipeline: Splitting and Transcoding

Source notebook

Repo path: gemini/use-cases/multimodal-data-curation/semantic-deduplication.ipynb · Open on GitHub · advanced

Deduplicates video clips using video embeddings and BigQuery vector search.

Summary

The notebook demonstrates a semantic deduplication stage for a multimodal video data curation pipeline. It loads video URIs from Cloud Storage, generates embeddings with Vertex AI Multimodal Embeddings or pooled SigLIP frame embeddings, stores them in BigQuery, creates a vector index, and removes near-duplicate records with VECTOR_SEARCH.

Key code patterns

Load GCS video URIs

def load_video_paths(bucket, num_videos):
    video_paths = []
    for i, blob in enumerate(storage_client.list_blobs(bucket)):
        if i >= num_videos:
            break
        video_paths.append(f"gs://{bucket}/" + blob.name)
    return video_paths

Builds the Cloud Storage URI list required by the Multimodal Embeddings API.

Vertex video embeddings

model = MultiModalEmbeddingModel.from_pretrained("multimodalembedding@001")
video = Video.load_from_file(video_file)
embeddings = model.get_embeddings(video=video)
row = {
    "uri": video_file,
    "embedding": embeddings.video_embeddings[0].embedding,
}

Generates semantic video embeddings directly from GCS-hosted video files.

Parallel embedding calls

semaphore = threading.Semaphore(max_workers)
with ThreadPoolExecutor(max_workers=max_workers) as executor:
    futures = [executor.submit(rate_limited_embedding_task, path) for path in video_files]
    for future in tqdm(as_completed(futures), total=len(video_files)):
        all_embeddings.append(future.result())

Improves throughput while limiting concurrent API requests.

Store embeddings in BigQuery

df_embedding = pd.DataFrame(embeddings)
pandas_gbq.to_gbq(
    df_embedding,
    f"{PROJECT_ID}.{dataset_id}.{table_id}",
    if_exists="replace",
    project_id=PROJECT_ID,
)

Persists embedding vectors so BigQuery can run vector search over them.

BigQuery vector dedupe

CREATE OR REPLACE TABLE '{full_table_id}_dedupe' AS
WITH dupes AS (
  SELECT DISTINCT query.uri
  FROM VECTOR_SEARCH(Table '{full_table_id}', "embedding", Table '{full_table_id}', top_k => 10)
  WHERE distance < 0.05 AND query.uri > base.uri
)
SELECT * FROM '{full_table_id}'
WHERE uri NOT IN (SELECT uri FROM dupes);

Uses cosine-distance nearest neighbors to identify and remove semantic duplicates.

Models & APIs used

  • Models: multimodalembedding@001, google/siglip2-base-patch16-512
  • APIs / services: Vertex AI, BigQuery, Cloud Storage
  • SDKs / libraries: vertexai, google-cloud-aiplatform, google-cloud-bigquery, google-cloud-storage, pandas-gbq, datasets, transformers, torch, PyAV, Pillow

When to use this

Use this pattern when curating a video dataset and you need a scalable, SQL-based semantic deduplication pass over embeddings.

Gotchas & caveats

  • Vertex AI API must be enabled for the selected Google Cloud project.
  • Videos must be stored in Cloud Storage for the Multimodal Embeddings API path shown.
  • LOCATION defaults to us-central1 from GOOGLE_CLOUD_REGION when set.
  • Parallel embedding calls may require a quota increase for better throughput.
  • Vertex multimodal video embeddings analyze only 2 minutes of content at a time and do not consider audio.
  • The distance threshold should be tuned for each dataset and use case.
  • The simple dedupe query ignores transitive links between near-neighbor records.
  • Vector index creation can take a few minutes and should be checked for 100 percent coverage before use.

Best practices

  • Store videos in Cloud Storage before calling the Multimodal Embeddings API.
  • Use a semaphore with ThreadPoolExecutor to control embedding API concurrency.
  • Capture errors per video URI instead of failing the whole embedding run.
  • Store embeddings in BigQuery for reproducible vector search workflows.
  • Create a BigQuery VECTOR INDEX with COSINE distance before large vector search workloads.
  • Monitor INFORMATION_SCHEMA.VECTOR_INDEXES coverage before relying on the index.
  • Tune top_k and distance_threshold for the dataset rather than treating defaults as universal.

Graph View

Backlinks