Generative AI Google Brain

Tuning & Customization - Best Practices

10 min read

Tuning & Customization — Best Practices

Distilled from 21 notebooks tagged Tuning & Customization in the GoogleCloudPlatform/generative-ai repository. The From the notebooks section below cites the per-notebook source for grounding.

Do this

  • Use high-quality, well-labeled, task-relevant data; low-quality data can hurt tuned model performance and introduce bias.
  • Create separate training, validation, and test or evaluation splits before tuning so reported quality is not measured on examples seen during training.
  • Store tuning datasets, validation data, evaluation outputs, OCR results, and model artifacts in Cloud Storage using clear or timestamped paths.
  • Validate input formats before launch: Gemini JSONL schemas, preference pairs, multimodal read configurations, image fileData references, embedding corpus/query/label files, TSV mappings, and prompt optimizer validation samples.
  • Establish a baseline before customization, then compare the base model, prompt, search app, or embedding model with the tuned or optimized result on the same task.
  • Choose metrics that match the task: ROUGE for summarization or captions, EM/F1 for Q&A, NDCG for embeddings, rubric metrics for model migration, custom metrics for task-specific SFT, and human plus automated ratings for prompts.
  • Start with default or recommended tuning and prompt optimizer settings, then adjust epochs, learning rate multiplier, adapter size, beta, generation parameters, or multi-metric settings only when the data and objective justify it.
  • Use validation data during tuning and store detailed row-level evaluation results in Cloud Storage for later inspection.
  • For classification-style evaluation, use low temperature and bounded output length to reduce response variability.
  • For preference optimization, validate chosen and rejected examples, filter incomplete rows, inspect score distributions, and keep pairs with clear quality separation.
  • For multimodal tuning, use GCS-hosted images, respect image count, size, and MIME constraints, attach read configurations, and run validity assessment before supervised fine-tuning.
  • For open-model tuning, prefer parameter-efficient techniques such as LoRA, 4-bit or 8-bit loading, gradient checkpointing, and appropriate GPU runtimes or managed containers.
  • Use secure storage or environment variables for provider keys and Hugging Face tokens instead of hardcoding secrets in notebooks.
  • Monitor asynchronous tuning, batch prediction, prompt optimization, and deployment jobs until terminal states before using their outputs.
  • Delete tuned endpoints, experiments, Cloud Storage artifacts, search engines, data stores, and temporary tutorial resources to avoid ongoing charges.

Avoid this

  • Launching jobs before enabling required APIs, billing, authentication, region, bucket, or service-account permissions.
  • Using local files where the managed service expects Cloud Storage URIs for training, validation, evaluation, translation TSV, or tuning artifacts.
  • Skipping train/validation/test splits or evaluating tuned outputs on data that was used for training.
  • Copying legacy Gemini hyperparameters directly to newer Gemini models instead of using current defaults or checking tuning data statistics.
  • Starting tuning with malformed data, such as missing preference fields, invalid GeminiExample turn order, empty splits, mismatched DOCX blocks, over-200-word translation lines, or incorrect image MIME/file references.
  • Assuming asynchronous jobs are ready immediately; tuning, evaluation, prompt optimization, endpoint deployment, and batch prediction can take minutes to hours.
  • Ignoring cost and cleanup: Vertex AI, Agent Platform, Cloud Storage, BigQuery, endpoints, GPUs, and search resources are billable.
  • Hardcoding API keys or tokens in notebook code instead of using environment variables or secure storage.

From the notebooks

Get started with embeddings tuning on Agent Platform

  • Prepare corpus, query, and labels files in the format required by embedding tuning.
  • Split labels into train and test sets before running the tuning pipeline.
  • Use a timestamped Cloud Storage path for tuning artifacts.
  • Evaluate tuned embeddings with pipeline-produced NDCG metrics.
  • Create cleanup flags for endpoint, model, job, bucket objects, and local tutorial files.

Migrating Foundation Models: A Practical Guide with Gen AI Evaluation Serivce

  • Use predefined rubric metrics such as GENERAL_QUALITY for structured model comparison.
  • Compare multiple candidates by passing a list of datasets to evaluate().
  • Use .show() on EvaluationDataset and EvaluationResult for in-notebook reports.
  • Use batch_evaluate() for large datasets or when immediate results are not required.
  • Use environment variables or secure storage for API keys instead of hardcoding them.

Intro to Agent Platform Multimodal Datasets

  • Use BigFrames instead of pandas for larger datasets.
  • Inspect created datasets with resource name, display name, BigQuery URI, and BigFrames preview.
  • Attach the read configuration to the dataset before tuning and batch prediction workflows.
  • Run tuning validity assessment before starting supervised fine-tuning.
  • Estimate tuning resources before running a tuning job.

Vertex Prompt Optimizer Notebook UI (Preview)

  • Use default optimization configurations as the initial setup.
  • Focus validation examples on the issues you want to address.
  • Use 50-100 distinct samples for reliable results.
  • Validate prompt and data before launching the optimizer job.
  • Use multi-metric settings only when more than one metric is needed.

Prepare High-Quality Preference Data for Gemini 2.5

  • Filter examples with missing prompts, chosen content, or rejected content before evaluation.
  • Prepare separate prompt-response dataframes for preferred and dispreferred responses with columns named prompt and response.
  • Sort evaluation case results by eval_case_index before attaching scores back to the dataframe.
  • Visualize preferred and dispreferred score distributions before choosing filtering thresholds.
  • Use win-margin filtering to keep pairs with clear quality separation.

Get Started with Gemini Preference Optimization

  • Validate chosen and rejected examples before writing tuning data.
  • Use JSONL with one Gemini preference example per line for training data.
  • Upload training and validation files to Cloud Storage before tuning.
  • Use a validation dataset with the tuning job.
  • Start with recommended hyperparameters and avoid beta=0 because it prevents learning.

Supervised Fine-Tuning with integrated Gen AI Evaluation

  • Use separate training and validation JSONL datasets for supervised tuning with integrated evaluation.
  • Store detailed row-level evaluation results in Cloud Storage using output_config.
  • Use custom model-based metrics with clear prompt templates and optional judge model system instructions.
  • Monitor asynchronous tuning jobs by refreshing job state until terminal completion.
  • Inspect checkpoint evaluation runs in Vertex AI Experiments to compare model progress over time.

Integrate Custom Metrics into Gemini Supervised Fine-Tuning

  • Track task-specific quality criteria during tuning instead of relying only on training loss.
  • Use validation data so custom metrics can be evaluated during training.
  • Aggregate custom metric scores with aggregation_metrics such as AVERAGE.
  • Store detailed evaluation results in GCS for later inspection.
  • Replace sample training and validation paths with production datasets.

Vertex AI SFT Gemini Migration Recipe

  • Do not apply the same hyperparameters from legacy Gemini models to latest Gemini models because model architecture and tuning infrastructure changed.
  • Use API or SDK defaults for new models when the legacy tuning job did not explicitly set hyperparameters.
  • Reuse the existing training and validation dataset URIs when creating the migrated tuning job.
  • Check tuning data statistics before changing epoch count for migration.

Supervised Fine Tuning with Gemini 2.5 Flash for Image Captioning

  • Use high-quality, well-labeled, task-relevant training data because low-quality data can hurt performance and introduce bias.
  • Use a separate validation set to evaluate model performance.
  • Choose evaluation metrics that reflect the task; this notebook uses ROUGE for image caption text generation.
  • Experiment with generation parameters and prompt structures to improve task performance.
  • Start with recommended default tuning hyperparameters, then customize epochs, learning rate multiplier, or adapter size for specific needs.

Supervised Fine Tuning with Gemini 2.0 Flash for change detection using the Google Gen AI SDK

  • Use JSON Lines format with one tuning example per line for Gemini supervised fine-tuning.
  • Copy tutorial data into your own Cloud Storage bucket before preparing tuning files.
  • Use default tuning settings for initial runs because the notebook says they are recommended for optimal performance.
  • Set epoch_count=1 in the tutorial to keep time and cost low.
  • Use temperature 0 when comparing a tuned model response with ground truth qualitatively.

Supervised Fine-tuning Gemini 2.5 Flash for Predictive Maintenance

  • Use train, validation, and test splits so evaluation samples are not seen during tuning.
  • Use Cloud Storage URIs for tuning datasets consumed by Vertex AI.
  • Set low temperature and max output tokens for classification-style tuned model evaluation.
  • Poll tuning job state until it leaves pending or running states before using the endpoint.
  • Check for empty data and failed job creation before launching or monitoring tuning.

Supervised fine-tuning with Gemini 2.0 Flash for Q&A using the Google Gen AI SDK

  • Establish a baseline with the default model before fine-tuning.
  • Use high-quality, well-labeled, task-relevant training data.
  • Keep test data formatted like training data to prevent training and serving skew.
  • Use system instructions to define desired model behavior and response style.
  • Use a separate evaluation set to assess model performance.

Supervised Fine Tuning with Gemini 2.5 Flash for Article Summarization

  • Use high-quality, well-labeled, task-relevant training data.
  • Use a separate validation or evaluation dataset to measure tuned model performance.
  • Evaluate the base model before tuning and the tuned endpoint after tuning.
  • Choose task-appropriate metrics; the notebook uses ROUGE-L for summarization.
  • Experiment with generation parameters, prompt structure, epochs, and learning rate multiplier.

Supervised Fine-tuning Gemini 2.5 Flash for Visual Defect Detection

  • Uses a separate train, validation, and test split with an 80/10/10 split.
  • Stores image files and JSONL tuning data in Cloud Storage for the fine-tuning service.
  • Uses guard clauses for missing data, unavailable endpoints, unexpected sample formats, and response parsing.
  • Sets a low temperature for deterministic classification during evaluation.
  • Uses a smaller adapter size for faster tuning in the demo.

MetaMath with Vertex AI Open Source Model Tuning

  • Use an 80/20 train-validation split with a fixed seed for reproducibility.
  • Limit validation rows to satisfy the Vertex AI validation dataset requirement.
  • Upload JSONL files to Cloud Storage because the Vertex AI tuning service cannot access local notebook files directly.
  • Use the same MetaMath prompt template for tuned-model testing and official-model comparison.
  • Use low temperature, top_p 1.0, and top_k 1 for factual math output.

Fine-tuning GPT-OSS 20B with Unsloth on Vertex AI Colab Enterprise and Nvidia A100 40GB GPU

  • Use 4-bit loading to reduce memory usage for the 20B model.
  • Use LoRA adapters for parameter-efficient fine-tuning instead of full fine-tuning.
  • Use Unsloth gradient checkpointing to reduce VRAM and support longer context.
  • Pin package versions and git revisions for compatibility in the managed notebook environment.
  • Track GPU memory before and after training.

Hugging Face DLCs: Fine-tuning Gemma with Transformer Reinforcement Learning (TRL) on Vertex AI

  • Use a Hugging Face PyTorch training DLC instead of manually building the training environment.
  • Use LoRA and an 8-bit optimizer to reduce VRAM requirements for fine-tuning.
  • Set output_dir under the /gcs mounted bucket path so trainer outputs are uploaded to Cloud Storage.
  • Pass logging environment variables so container logs stream cleanly to Google Cloud Logs Explorer.
  • Use a read-only or fine-grained Hugging Face token for gated model access.

Search tuning in Vertex AI Search

  • Prepare datasets in JSONL format with identifier-text pairs.
  • Represent query-to-answer mappings in tab-separated TSV files.
  • Upload additional tuning documents and datasets to Cloud Storage before datastore refresh and tuning.
  • Use INCREMENTAL document import when adding files to the datastore.
  • Keep a baseline search result before tuning so it can be compared with post-tuning behavior.

Tutorial for Running Prompt Management and Evaluation

  • Version prompts each time changes are saved so iterations can be compared.
  • Test a prompt with sample input before saving and evaluating it.
  • Create datasets in Cloud Storage and upload CSV, JSON, or JSONL files for evaluation.
  • Run an initial evaluation before prompt changes to establish a baseline.
  • Use human-in-the-loop rating and automated metrics for evaluation.

Generate training dataset for Cloud Translation API NMT (Neural Machine Translation) model training

  • Remove empty lines from both source and reference documents before making line pairs.
  • Preserve document order across paragraphs and tables when generating training pairs.
  • Validate source and reference paths and formats before processing.
  • Capture mismatched blocks and over-200-word lines instead of silently adding bad TSV rows.
  • Use dataset display names built from prefix, source language, target language, and optional suffix.

Back to Tuning & Customization · Best Practices Map

Graph View

Backlinks