Generative AI Google Brain

gemini-tuning-sft-gemini-custom-metric-evaluation

3 min read

Integrate Custom Metrics into Gemini Supervised Fine-Tuning

Source notebook

Repo path: gemini/tuning/sft_gemini_custom_metric_evaluation.ipynb · Open on GitHub · intermediate

Adds a Python custom metric to Gemini SFT using Vertex AI REST tuning jobs and GCS outputs.

Summary

This notebook teaches how to define a custom Python evaluation metric for Gemini supervised fine-tuning and attach it to a Vertex AI tuning job. It builds a word-overlap F1 scorer for summarization, embeds it in evaluationConfig, submits the job with curl against the Vertex AI REST API, then checks job status and lists evaluation results in Cloud Storage.

Key code patterns

Custom evaluate function

evaluation_function = '''def evaluate(instance):
    prediction = instance.get("prediction", "").strip().lower()
    reference = instance.get("reference", "").strip().lower()
    if not prediction or not reference:
        return 0.0
    if prediction == reference:
        return 1.0
    pred_words = set(prediction.split())
    ref_words = set(reference.split())
    overlap = pred_words.intersection(ref_words)
    precision = len(overlap) / len(pred_words)
    recall = len(overlap) / len(ref_words)
    return 2 * precision * recall / (precision + recall) if precision + recall else 0.0
'''

The tuning service expects a string-defined function named evaluate that accepts prediction and reference data.

Attach metric to tuning job

tuning_request = {
  "base_model": "gemini-2.5-flash",
  "supervisedTuningSpec": {
    "trainingDatasetUri": TRAINING_DATASET_URI,
    "validationDatasetUri": VALIDATION_DATASET_URI,
    "evaluationConfig": {
      "metrics": {
        "aggregation_metrics": ["AVERAGE"],
        "custom_code_execution_spec": {"evaluation_function": evaluation_function}
      }
    }
  }
}

evaluationConfig is the integration point for custom code execution and metric aggregation during SFT.

Save evaluation output to GCS

EVAL_OUTPUT_URI = f"{BUCKET_URI}/evaluation_results"
"outputConfig": {
  "gcs_destination": {
    "output_uri_prefix": EVAL_OUTPUT_URI
  }
}

Detailed per-example scores and aggregate metrics are written to Cloud Storage.

Submit with REST API

API_ENDPOINT = f"https://{LOCATION}-aiplatform.googleapis.com/v1beta1/projects/{PROJECT_ID}/locations/{LOCATION}/tuningJobs"
!curl -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  {API_ENDPOINT} \
  -d @tuning_request.json

The notebook uses the Vertex AI REST API directly instead of an AI SDK.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI, Vertex AI Gen AI Evaluation service, Cloud Storage

When to use this

Use this pattern when Gemini SFT needs task-specific validation metrics beyond training loss, such as summary quality scoring.

Gotchas & caveats

  • A Google Cloud project with billing enabled is required.
  • The Vertex AI API must be enabled.
  • A Google Cloud Storage bucket is required for evaluation output.
  • Training and validation files must already be in GCS and use supervised tuning JSONL format.
  • The custom metric must be a string function named evaluate that accepts one instance argument.
  • The notebook uses the v1beta1 Vertex AI tuningJobs REST endpoint.
  • The example expects an access token from gcloud auth print-access-token.
  • Training is described as running in the background for 30-60 minutes.

Best practices

  • Track task-specific quality criteria during tuning instead of relying only on training loss.
  • Use validation data so custom metrics can be evaluated during training.
  • Aggregate custom metric scores with aggregation_metrics such as AVERAGE.
  • Store detailed evaluation results in GCS for later inspection.
  • Replace sample training and validation paths with production datasets.
  • Compare multiple tuning jobs with different configurations and metrics.

Graph View

Backlinks