Generative AI Google Brain

gemini-tuning-dpo-gemini-get-started

3 min read

Get Started with Gemini Preference Optimization

Source notebook

Repo path: gemini/tuning/dpo_gemini_get_started.ipynb · Open on GitHub · intro

Tunes Gemini 2.5 Flash with human preference data using Vertex AI preference optimization.

Summary

This notebook teaches preference optimization by converting human-preferred and rejected responses into Gemini tuning format. It loads UltraFeedback data, writes JSONL training and validation files, uploads them to Cloud Storage, submits a Vertex AI preference tuning job, checks job status, and compares base versus tuned model outputs.

Key code patterns

Initialize Vertex Gemini client

from google import genai
 
LOCATION = "us-central1"
client = genai.Client(
    vertexai=True,
    project=PROJECT_ID,
    location=LOCATION,
)

Connects google-genai to Vertex AI in the configured Google Cloud project and region.

Convert preferences to Gemini JSONL

return {
    "contents": [{"role": "user", "parts": [{"text": user_prompt}]}],
    "completions": [
        {"score": 1.0, "completion": {"role": "model", "parts": [{"text": chosen_response}] }},
        {"score": 0.0, "completion": {"role": "model", "parts": [{"text": rejected_response}] }},
    ],
}

Maps chosen and rejected UltraFeedback responses into the preference tuning format expected by Gemini.

Submit preference tuning job

tuning_job = client.tunings.tune(
    base_model="gemini-2.5-flash",
    training_dataset=types.TuningDataset(gcs_uri=TRAIN_URI),
    config=types.CreateTuningJobConfig(
        method="PREFERENCE_TUNING",
        beta=BETA,
        validation_dataset=types.TuningDataset(gcs_uri=VAL_URI),
    ),
)

Starts an asynchronous Vertex AI tuning job from Cloud Storage JSONL data.

Generate with tuned endpoint

tuned_response = client.models.generate_content(
    model=tuning_job.tuned_model.endpoint,
    contents=test_prompt,
    config={
        "thinking_config": {"thinking_budget": 0},
        "max_output_tokens": 1024,
        "temperature": 1.0,
    },
)

Uses the tuned model endpoint as the model name for post-training comparison.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI, Cloud Storage
  • SDKs / libraries: google-genai, google-cloud-aiplatform, datasets

When to use this

Use this pattern when you have paired preferred and rejected responses and want Gemini outputs to better match human preferences.

Gotchas & caveats

  • Requires a Google Cloud project with billing enabled.
  • Requires the Vertex AI API to be enabled.
  • Colab users must authenticate with google.colab.auth.authenticate_user().
  • PROJECT_ID must be set or available through GOOGLE_CLOUD_PROJECT.
  • The notebook uses us-central1 and states it has the best availability.
  • Cloud Storage bucket names must be globally unique.
  • Colab may require a runtime restart after installing packages.
  • Tuning is asynchronous and the notebook estimates 30-60 minutes for training.

Best practices

  • Validate chosen and rejected examples before writing tuning data.
  • Use JSONL with one Gemini preference example per line for training data.
  • Upload training and validation files to Cloud Storage before tuning.
  • Use a validation dataset with the tuning job.
  • Start with recommended hyperparameters and avoid beta=0 because it prevents learning.
  • Compare base and tuned responses on the same prompt.
  • Delete the tuned endpoint to stop billing for the deployed model.

Graph View

Backlinks