Generative AI Google Brain

gemini-multimodal-dataset-intro-vertex-ai-multimodal-dataset

3 min read

Intro to Agent Platform Multimodal Datasets

Source notebook

Repo path: gemini/multimodal-dataset/intro_vertex_ai_multimodal_dataset.ipynb · Open on GitHub · intermediate

Builds Agent Platform multimodal datasets for Gemini tuning validation, resource estimates, tuning, and batch prediction.

Summary

This notebook teaches how to create Agent Platform multimodal datasets from flower image URIs and labels, attach a Gemini request template, and assemble examples into BigQuery. It demonstrates validation for supervised fine-tuning, tuning resource estimation, starting a Gemini tuning job, polling job state, and running batch prediction from an assembled dataset.

Key code patterns

Initialize clients

PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")
LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "us-central1")
client = vertexai.Client(project=PROJECT_ID, location=LOCATION)
bpd.options.bigquery.project = PROJECT_ID
bpd.options.bigquery.location = LOCATION

Sets Agent Platform and BigFrames to the same project and region.

Create multimodal dataset

flowers = client.datasets.create_from_pandas(dataframe=training_set)
show_dataset_info(flowers)
flowers.to_bigframes().head()

Creates a managed multimodal dataset from image URI and label columns.

Template Gemini requests

read_config = types.GeminiRequestReadConfig.single_turn_template(
    prompt="This is the image: {image_uris}",
    response="{labels}",
    system_instruction="You are a botanical image classifier."
)

Maps dataset columns into Gemini request contents and expected responses.

Assemble dataset

assembly_bq_uri = client.datasets.assemble(
    name=flowers.name,
    gemini_request_read_config=read_config,
)
bpd.read_gbq_table(assembly_bq_uri.removeprefix("bq://")).head()

Materializes assembled Gemini requests into a BigQuery table for inspection.

Validate for tuning

validation = client.datasets.assess_tuning_validity(
    dataset_name=flowers.name,
    model_name="gemini-2.0-flash-001",
    dataset_usage="SFT_TRAINING",
)
validation.errors

Checks whether assembled examples are valid for supervised fine-tuning.

Tune with dataset resource

genai_client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)
tuning_job = genai_client.tunings.tune(
    base_model="gemini-2.5-flash",
    training_dataset={"vertex_dataset_resource": flowers.name},
    config=genai_types.CreateTuningJobConfig(
        validation_dataset={"vertex_dataset_resource": flowers_validation.name}
    ),
)

Starts Gemini tuning using Agent Platform dataset resources.

Batch prediction

job = genai_client.batches.create(
    model=model,
    src=prediction_assembly_bq_uri,
)
while job.state not in completed_states:
    time.sleep(30)
    job = genai_client.batches.get(name=job.name)

Runs batch prediction from the assembled BigQuery URI and polls completion.

Models & APIs used

When to use this

Use this pattern when preparing multimodal image-label data for Gemini supervised fine-tuning and batch prediction on Vertex AI.

Gotchas & caveats

  • Billing must be enabled for Agent Platform, Cloud Storage, and BigQuery.
  • The Agent Platform API must be enabled for the Google Cloud project.
  • Local environments need Application Default Credentials from gcloud auth application-default login.
  • The default region is us-central1 unless GOOGLE_CLOUD_REGION is set.
  • Prediction size and per-category train-validation counts must not exceed available examples.
  • Tuning requires an attached read configuration on the multimodal dataset.
  • The Agent Platform tuning service account needs read permissions on referenced GCS buckets.
  • Invalid GeminiExample turn order, such as consecutive user roles, produces validation errors.

Best practices

  • Use BigFrames instead of pandas for larger datasets.
  • Inspect created datasets with resource name, display name, BigQuery URI, and BigFrames preview.
  • Attach the read configuration to the dataset before tuning and batch prediction workflows.
  • Run tuning validity assessment before starting supervised fine-tuning.
  • Estimate tuning resources before running a tuning job.
  • Use separate training, validation, and prediction subsets.
  • Poll tuning and batch jobs until terminal states before using their outputs.

Graph View

Backlinks