Generative AI Google Brain

Tuning & Customization

6 min read

Tuning & Customization

Tuning & Customization on Google Cloud covers workflows that adapt foundation models, prompts, embeddings, search ranking, and translation models to task-specific data. In these notebooks, customization is driven by curated datasets, Cloud Storage artifacts, Vertex AI or Agent Platform jobs, and evaluation before and after changes.

Synthesized across 21 notebooks. See the source pages under Notebooks below for line-level grounding.

This area spans prompt optimization, supervised fine-tuning, preference optimization, embedding tuning, Vertex AI Search tuning, open-model tuning, and Cloud Translation custom NMT training. The common pattern is to prepare task-specific data in the required format, store it in Cloud Storage, run a managed tuning or optimization job, and evaluate the result against a baseline.

Most Gemini workflows use JSONL training and validation files, separate evaluation or test splits, and asynchronous Vertex AI tuning jobs that may automatically deploy tuned models to endpoints. Prompt and migration workflows emphasize evaluation datasets, rubric or custom metrics, and re-evaluation after prompt or model changes.

Multimodal and open-model workflows add extra operational requirements: GCS file references for images, attached read configurations for Agent Platform multimodal datasets, GPU-specific notebook runtimes or custom containers, Hugging Face tokens for gated models, and explicit cleanup to avoid endpoint, storage, and experiment charges.

Canonical workflow

  1. Enable the required APIs, configure project, region, billing, authentication, service accounts, and Cloud Storage buckets.
  2. Prepare task-specific data in the required format, such as Gemini JSONL, preference JSONL, embedding corpus/query/label files, TSV mappings, CSV or JSONL validation samples, or DOCX-derived translation TSV rows.
  3. Split data into train, validation, and test or evaluation sets before tuning, and keep serving or test examples formatted like training data.
  4. Establish a baseline with the base model, existing prompt, search app, embedding model, or translation dataset before customization.
  5. Validate data schemas, sample counts, multimodal turn order, labels, targets, GCS permissions, and job configuration before launching tuning or optimization.
  6. Run the tuning, prompt optimization, evaluation, batch prediction, or custom container job, then poll asynchronous jobs until they reach a terminal state.
  7. Evaluate the tuned or optimized result with task-appropriate metrics, row-level outputs, generated responses, or before-and-after comparisons against the baseline.
  8. Inspect artifacts, experiment runs, endpoints, and reports, then clean up endpoints, buckets, jobs, experiments, data stores, engines, and local files when no longer needed.

Best practices

See Tuning & Customization - Best Practices for the full distilled guide.

  • Use high-quality, well-labeled, task-relevant data; low-quality data can hurt tuned model performance and introduce bias.
  • Create separate training, validation, and test or evaluation splits before tuning so reported quality is not measured on examples seen during training.
  • Store tuning datasets, validation data, evaluation outputs, OCR results, and model artifacts in Cloud Storage using clear or timestamped paths.
  • Validate input formats before launch: Gemini JSONL schemas, preference pairs, multimodal read configurations, image fileData references, embedding corpus/query/label files, TSV mappings, and prompt optimizer validation samples.
  • Establish a baseline before customization, then compare the base model, prompt, search app, or embedding model with the tuned or optimized result on the same task.
  • Choose metrics that match the task: ROUGE for summarization or captions, EM/F1 for Q&A, NDCG for embeddings, rubric metrics for model migration, custom metrics for task-specific SFT, and human plus automated ratings for prompts.

Key models & APIs

  • gemini-2.0-flash, gemini-2.0-flash-001, gemini-2.0-flash-lite-001, gemini-2.5-flash, gemini-2.5-flash-001, gemini-2.5-flash-lite, gemini-2.5-pro, gemini-1.5-flash-002, gemini-1.5-pro-002, text-embedding-005, gpt-4o, meta/llama3_1@llama-3.1-8b, meta-math/MetaMath-7B-V1.0, unsloth/gpt-oss-20b, google/gemma-2b, Vertex AI API, Agent Platform API, Vertex AI Gen AI Evaluation SDK and Service, Google Gen AI SDK, Vertex AI Prompt Optimizer, Vertex AI Experiments, Vertex AI tuningJobs REST v1beta1, Vertex AI Search and Discovery Engine, Document AI API, Cloud Translation API custom NMT training, Cloud Storage and Cloud Storage API, BigQuery and BigFrames, Artifact Registry API, Hugging Face PyTorch training DLCs, TRL, Unsloth

Notebooks (21)

Embeddings & Vector Search (1)

Gemini (14)

Open Models (3)

Vertex AI Search (1)

Tools (1)

Translation (1)

Part of the Generative AI Google Brain · Tuning & Customization - Best Practices

Graph View

Backlinks