Generative AI Google Brain

gemini-tuning-sft-gemini-automatic-evaluation

3 min read

Supervised Fine-Tuning with integrated Gen AI Evaluation

Source notebook

Repo path: gemini/tuning/sft_gemini_automatic_evaluation.ipynb · Open on GitHub · intermediate

Fine-tunes gemini-2.5-flash with automatic Gen AI Evaluation metrics at each checkpoint.

Summary

This notebook teaches how to configure integrated evaluation for supervised fine-tuning in Vertex AI. It creates training and validation dataset references, defines an EvaluationConfig with a custom fluency metric, launches a Gemini tuning job, monitors completion, and retrieves checkpoint metrics from Vertex AI Experiments. It also includes an optional Gradio viewer for row-level evaluation results stored in Cloud Storage.

Key code patterns

Initialize Vertex AI and Gen AI SDK

aiplatform.init(project=PROJECT_ID, location=LOCATION, staging_bucket=BUCKET_URI)
client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)

Sets project, region, staging bucket, and Vertex AI-backed Gemini client for tuning.

Define EvaluationConfig

evaluation_config = types.EvaluationConfig(
    metrics=[types.Metric(name="fluency", prompt_template="Evaluate the sentence fluency of the response. Provide a score from 1-5.\n RESPONSE: {response}")],
    output_config=types.OutputConfig(gcs_destination=types.GcsDestination(output_uri_prefix=f"{BUCKET_URI}/evaluation")),
    autorater_config=types.AutoraterConfig(sampling_count=6),
)

Attaches model-based checkpoint evaluation and writes detailed results to Cloud Storage.

Launch supervised tuning

sft_tuning_job = client.tunings.tune(
    base_model=base_model,
    training_dataset={"gcs_uri": training_dataset_uri},
    config=types.CreateTuningJobConfig(
        tuned_model_display_name=tuned_model_display_name,
        validation_dataset=validation_dataset,
        evaluation_config=evaluation_config,
    ),
)

Starts an asynchronous SFT job with validation data and integrated evaluation enabled.

Read checkpoint metrics

experiment_runs = aiplatform.ExperimentRun.list(experiment=experiment_name)
for run in experiment_runs:
    if "-evaluation-" in run.name:
        print(pd.DataFrame.from_dict(run.get_metrics(), orient="index"))

Retrieves aggregated evaluation metrics from Vertex AI Experiment runs.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI, Gen AI Evaluation Service, Cloud Storage, Vertex AI Experiments
  • SDKs / libraries: google-genai, google-cloud-aiplatform, google-cloud-storage, gradio, plotly, pandas

When to use this

Use this pattern when fine-tuning Gemini and needing checkpoint-level quality metrics during training.

Gotchas & caveats

  • A Google Cloud project with the Vertex AI API enabled is required.
  • A Cloud Storage bucket is required for tuning and evaluation artifacts.
  • The notebook defaults LOCATION to us-central1 if GOOGLE_CLOUD_REGION is not set.
  • Training and evaluation are asynchronous and the notebook warns completion takes about 45 minutes.
  • The SDK tuning implementation is marked experimental and may change in future versions.
  • Higher autorater sampling_count can improve metrics but increases cost and time.

Best practices

  • Use separate training and validation JSONL datasets for supervised tuning with integrated evaluation.
  • Store detailed row-level evaluation results in Cloud Storage using output_config.
  • Use custom model-based metrics with clear prompt templates and optional judge model system instructions.
  • Monitor asynchronous tuning jobs by refreshing job state until terminal completion.
  • Inspect checkpoint evaluation runs in Vertex AI Experiments to compare model progress over time.
  • Clean up experiments, endpoints, and Cloud Storage artifacts to avoid unexpected charges.

Graph View

Backlinks