Generative AI Google Brain

gemini-tuning-sft-gemini-visual-defect-detection

3 min read

Supervised Fine-tuning Gemini 2.5 Flash for Visual Defect Detection

Source notebook

Repo path: gemini/tuning/sft_gemini_visual_defect_detection.ipynb · Open on GitHub · intermediate

Fine-tunes Gemini 2.5 Flash on image-based manufacturing defect classification.

Summary

This notebook teaches supervised fine-tuning of Gemini 2.5 Flash for a multimodal visual defect detection task. It generates simulated product images, uploads them to Cloud Storage, builds Gemini tuning JSONL with text and image parts, launches and monitors a Vertex AI tuning job, evaluates the tuned endpoint qualitatively, and uses Gemini to summarize the job result.

Key code patterns

Initialize Vertex AI and GenAI client

vertexai.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)
vertex_client = VertexClient(
    vertexai=True,
    project=PROJECT_ID,
    location=REGION,
)

Configures Vertex AI staging and the google-genai client used for tuning jobs and predictions.

Build multimodal tuning example

return {
  "contents": [
    {"role": "user", "parts": [
      {"text": prompt},
      {"fileData": {"mimeType": "image/png", "fileUri": image_uri}},
    ]},
    {"role": "model", "parts": [{"text": label}]},
  ]
}

Shows the JSONL structure for supervised tuning with text plus image input and a labeled model response.

Launch supervised tuning

sft_tuning_job = vertex_client.tunings.tune(
    base_model=BASE_MODEL_ID,
    training_dataset={"gcs_uri": TRAIN_JSONL_GCS_URI},
    config=genai_types.CreateTuningJobConfig(
        adapter_size="ADAPTER_SIZE_FOUR",
        epoch_count=5,
        tuned_model_display_name=TUNED_MODEL_DISPLAY_NAME,
        validation_dataset=validation_dataset,
    ),
)

Starts a Vertex AI supervised fine-tuning job from Cloud Storage training and validation files.

Predict with tuned endpoint

response = vertex_client.models.generate_content(
    model=tuned_endpoint,
    contents=prediction_contents,
    config={"temperature": 0.1, "max_output_tokens": 2000},
)
predicted_output = _extract_predicted_text(response)

Uses the tuned model endpoint for low-temperature multimodal classification on held-out test images.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI, Cloud Storage
  • SDKs / libraries: google-genai, vertexai, google-cloud-storage, pandas, gcsfs, Pillow

When to use this

Use this pattern when adapting Gemini to a specialized image classification and defect-description workflow with labeled multimodal examples.

Gotchas & caveats

  • Requires PROJECT_ID, REGION, and a Cloud Storage bucket before Vertex AI operations can run.
  • The notebook says to restart the kernel after installing packages.
  • Training, validation, and test JSONL files must be accessible in Cloud Storage.
  • Pandas GCS writes require credentials passed through storage_options with gcsfs installed.
  • Fine-tuning can take 30 minutes to several hours depending on dataset size, base model, and adapter size.
  • Evaluation is qualitative and prints comparisons rather than computing aggregate metrics.

Best practices

  • Uses a separate train, validation, and test split with an 80/10/10 split.
  • Stores image files and JSONL tuning data in Cloud Storage for the fine-tuning service.
  • Uses guard clauses for missing data, unavailable endpoints, unexpected sample formats, and response parsing.
  • Sets a low temperature for deterministic classification during evaluation.
  • Uses a smaller adapter size for faster tuning in the demo.
  • Checks for an existing tuning job with the same display name after a launch error.

Graph View

Backlinks