Generative AI Google Brain

gemini-tuning-sft-gemini-on-image-captioning

3 min read

Supervised Fine Tuning with Gemini 2.5 Flash for Image Captioning

Source notebook

Repo path: gemini/tuning/sft_gemini_on_image_captioning.ipynb · Open on GitHub · intermediate

Fine-tunes Gemini 2.5 Flash on GCS image-caption pairs and evaluates ROUGE before and after tuning.

Summary

This notebook teaches supervised fine-tuning of gemini-2.5-flash for image captioning with Vertex AI. It prepares multimodal JSONL training and validation data using GCS image URIs, evaluates the base model with ROUGE, launches a tuning job, waits for completion, and evaluates the tuned endpoint. It also shows prediction, result export, and cleanup of experiments, endpoints, and the Cloud Storage bucket.

Key code patterns

Initialize clients

aiplatform.init(project=PROJECT_ID, location=REGION)
client = genai.Client(vertexai=True, project=PROJECT_ID, location=REGION)

Sets the project and region for Vertex AI and creates a Gen AI SDK client targeting Vertex AI.

Create multimodal tuning rows

instance = {
  "contents": [
    {"role": "user", "parts": [
      {"fileData": {"mimeType": "image/jpeg", "fileUri": f"{BUCKET_URI}/images/{obj['image']}"}},
      {"text": task_prompt}
    ]},
    {"role": "model", "parts": [{"text": obj["suffix"]}]}
  ]
}

Formats each image-caption pair as single-turn Gemini tuning data with GCS fileData input and text output.

Generate from image URI

response = client.models.generate_content(
    model=base_model,
    contents=[
        types.Part.from_uri(file_uri=str(query_image_uri), mime_type="image/jpeg"),
        task_prompt,
    ],
    config={"temperature": 0.0},
)
prediction = response.text.strip()

Uses the same image URI and prompt format for baseline and tuned-model inference.

Launch supervised tuning

sft_tuning_job = client.tunings.tune(
    base_model=base_model,
    training_dataset={"gcs_uri": f"{BUCKET_URI}/train/train.jsonl"},
    config=types.CreateTuningJobConfig(
        adapter_size="ADAPTER_SIZE_EIGHT",
        epoch_count=1,
        tuned_model_display_name=tuned_model_display_name,
        validation_dataset=validation_dataset,
    ),
)

Creates a Vertex AI supervised tuning job with training data, validation data, adapter size, and epoch count.

Compute ROUGE metrics

scorer = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeL"], use_stemmer=True)
scores = scorer.score(target=groundTruth, prediction=prediction)
metrics = {
  "rougeL_precision": scores.get("rougeL").precision,
  "rougeL_recall": scores.get("rougeL").recall,
  "rougeL_fmeasure": scores.get("rougeL").fmeasure,
}

Compares generated captions against validation ground truth before and after tuning.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI, Cloud Storage
  • SDKs / libraries: google-genai, google-cloud-aiplatform, google-cloud-storage, jsonlines, pandas, Pillow, rouge_score, tqdm

When to use this

Use this pattern when you need to adapt Gemini 2.5 Flash to a labeled image-to-text task and compare base versus tuned output quality.

Gotchas & caveats

  • Vertex AI Workbench is already authenticated, but local Jupyter may require gcloud auth login and Colab requires auth.authenticate_user().
  • Tuning examples support images and text as input but text only as output.
  • Image inputs must be GCS fileData references, not inline data.
  • Each tuning example supports up to 30 images, each image up to 20MB, with image/jpeg or image/png MIME types.
  • The notebook uses billable Vertex AI and Cloud Storage resources.
  • The tuning job is expected to take 30-40 minutes with the provided dataset and settings.
  • The evaluation loop sleeps between calls and retries after a 30 second delay on errors.
  • The tuning job automatically creates and deploys to a Vertex AI endpoint that should be deleted during cleanup.

Best practices

  • Use high-quality, well-labeled, task-relevant training data because low-quality data can hurt performance and introduce bias.
  • Use a separate validation set to evaluate model performance.
  • Choose evaluation metrics that reflect the task; this notebook uses ROUGE for image caption text generation.
  • Experiment with generation parameters and prompt structures to improve task performance.
  • Start with recommended default tuning hyperparameters, then customize epochs, learning rate multiplier, or adapter size for specific needs.
  • Set epoch_count=1 when keeping tuning time and cost low is important.
  • Delete experiments, endpoints, and Cloud Storage resources when cleanup is needed.

Graph View

Backlinks