Generative AI Google Brain

gemini-tuning-sft-gemini-on-multiple-images

3 min read

Supervised Fine Tuning with Gemini 2.0 Flash for change detection using the Google Gen AI SDK

Source notebook

Repo path: gemini/tuning/sft_gemini_on_multiple_images.ipynb · Open on GitHub · intermediate

Fine-tunes Gemini 2.0 Flash on paired images for spot-the-difference change detection.

Summary

This notebook teaches supervised fine-tuning of Gemini 2.0 Flash on multimodal JSONL examples that contain two image inputs plus text. It copies a Spot-the-diff-style dataset to Cloud Storage, converts train and validation JSONL rows into Gemini tuning format, starts a Vertex AI supervised tuning job with the Google Gen AI SDK, then qualitatively tests the tuned endpoint on a random image pair.

Key code patterns

Initialize Vertex AI GenAI client

vertexai.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)
client = genai.Client(vertexai=True, project=PROJECT_ID, location=REGION)

Connects the Google Gen AI SDK to Vertex AI using the selected project, region, and staging bucket.

Build multimodal tuning rows

instance = {
  "systemInstruction": {"role": "string", "parts": [{"text": SYSTEM_INSTRUCTION}]},
  "contents": [{"role": "user", "parts": [
    {"fileData": {"mimeType": "image/png", "fileUri": f"{image_path}.png"}},
    {"text": "Image 1."},
    {"fileData": {"mimeType": "image/png", "fileUri": f"{image_path}_2.png"}},
    {"text": TASK_PROMPT}]},
    {"role": "model", "parts": [{"text": obj["sentences"]}]}]
}

Shows the JSONL structure for supervised tuning with multiple image inputs and a target text answer.

Create tuning job

sft_tuning_job = client.tunings.tune(
    base_model=MODEL_ID,
    training_dataset={"gcs_uri": train_dataset},
    config=types.CreateTuningJobConfig(
        adapter_size="ADAPTER_SIZE_EIGHT",
        epoch_count=1,
        tuned_model_display_name=tuned_model_display_name,
    ),
)

Starts supervised fine-tuning with default-oriented settings and one epoch to reduce time and cost.

Use tuned endpoint

contents = [
  "Image 1:", types.Part.from_uri(file_uri=str(input_image_one_uri), mime_type="image/jpeg"),
  "Image 2:", types.Part.from_uri(file_uri=str(input_image_two_uri), mime_type="image/jpeg"),
]
response = client.models.generate_content(
    model=tuning_job.tuned_model.endpoint,
    contents=contents,
    config={"temperature": 0},
)

Runs deterministic qualitative evaluation against the tuned model endpoint.

Models & APIs used

  • Models: gemini-2.0-flash-001
  • APIs / services: Vertex AI, Cloud Storage
  • SDKs / libraries: google-genai, google-cloud-aiplatform, vertexai, etils, Pillow, matplotlib, numpy

When to use this

Use this pattern when you need to adapt Gemini to a specialized vision task involving paired or multiple image inputs and text outputs.

Gotchas & caveats

  • Requires an existing Google Cloud project with the Vertex AI API enabled.
  • Requires authentication in Colab via google.colab.auth.authenticate_user().
  • Requires a Cloud Storage bucket and gs:// dataset paths accessible from the project.
  • The notebook states package installation requires a runtime restart.
  • The tuning job is expected to take about 30 minutes with the provided dataset and settings.
  • The validation_dataset object is created, but the shown tune call only passes training_dataset.
  • The validation file cell saves train_instances to prepared_val.jsonl instead of val_instances.
  • Training examples use image/png MIME type, while inference uses image/jpeg MIME type for .png paths.

Best practices

  • Use JSON Lines format with one tuning example per line for Gemini supervised fine-tuning.
  • Copy tutorial data into your own Cloud Storage bucket before preparing tuning files.
  • Use default tuning settings for initial runs because the notebook says they are recommended for optimal performance.
  • Set epoch_count=1 in the tutorial to keep time and cost low.
  • Use temperature 0 when comparing a tuned model response with ground truth qualitatively.

Graph View

Backlinks