Generative AI Google Brain

Vision - Best Practices

17 min read

Vision — Best Practices

Distilled from 45 notebooks tagged Vision in the GoogleCloudPlatform/generative-ai repository. The From the notebooks section below cites the per-notebook source for grounding.

Do this

  • Pass every non-text input with the correct MIME type, using supported structures such as Part.from_uri, Part.from_bytes, fileData, Image.from_file, gcs_uri images, or BigQuery ObjectRefs.
  • Use response_mime_type=“application/json”, response_schema, Pydantic models, enums, and custom parsers whenever downstream code depends on model output.
  • Set temperature to 0 or a low value for deterministic extraction, classification, scene-boundary detection, attribute labeling, and evaluation comparisons.
  • Ground prompts explicitly: tell the model to use only the provided media or retrieved context, label candidate images, constrain answers to visible evidence, and request no preamble when only the answer is needed.
  • Prototype with a single image, video, document, or prompt before launching batch prediction, tuning, vector indexing, or large generation jobs.
  • Inspect intermediate artifacts: response parts, generated code execution output, parsed JSON, rubrics, validator questions, row-level explanations, visual overlays, captions, and side-by-side edited images.
  • Throttle high-volume embedding, generation, and batch workflows; use semaphores or thread pools, moderate batch sizes, checkpoint to Cloud Storage, log per-item errors, and monitor quota usage.
  • Store traceability metadata with outputs, including evaluation IDs, timestamps, prompt text, GCS URIs, model version, usage metadata, token counts, source IDs, and generated asset metadata.
  • Use BigQuery remote models, AI functions, views, joins, VECTOR_SEARCH, and vector indexes when the data already lives in BigQuery or when reproducible evaluation tables are needed.
  • Use generated rubrics, custom metrics, counterexample prompts, reference ground truth, DeepDiff-style field comparisons, and manual question edits to validate multimodal evaluation quality.
  • For supervised fine-tuning, use high-quality task-relevant labels, separate train/validation/test splits, Cloud Storage JSONL files, conservative default hyperparameters, and low-temperature evaluation.
  • For image and video generation, set safety_filter_level, person_generation, aspect ratio, image count, resolution, duration, audio generation, output MIME type, and output_gcs_uri explicitly.
  • For visual consistency and editing, use reference images, character sheets, blank canvases for aspect ratio control, precise object-change prompts, masks where required, and saved source IDs or prompts.
  • For video curation, filter by metadata before expensive processing, tune scene-detection and quality thresholds on samples, use FFmpeg/ffprobe carefully, and remember that multimodal video embeddings in the shown workflow do not consider audio.
  • Clean up billable resources such as buckets, BigQuery datasets, remote models, Vector Search indexes, endpoints, experiments, Cloud Run services, and deployed tuning endpoints.

Avoid this

  • Running notebooks before enabling the required API, billing, authentication, project ID, region, or service-account IAM roles.
  • Confusing Vertex AI and Agent Platform setup requirements, or missing Colab-specific authentication and runtime restarts after package installation.
  • Using the wrong location: several notebooks default to us-central1, some use global, preview models may require global, and BigQuery dataset location must match the Vertex AI batch location.
  • Exceeding quota with parallel embedding, batch, tuning, or generation workflows, causing ResourceExhausted errors or 429 retries.
  • Passing media in an unsupported form, with the wrong MIME type, inaccessible Cloud Storage URI, hard-coded image/jpeg assumption, or inline data where GCS fileData is required.
  • Trusting free-form JSON, exact-match JSON formatting, generated rubrics, validator scores, or autorater explanations without custom parsing and manual inspection.
  • Assuming prompts, aesthetic thresholds, vector distance thresholds, scene-detection thresholds, and bounding-box behavior generalize across datasets without tuning.
  • Starting long-running or billable jobs without polling operation state, setting output destinations, estimating costs, or deleting created resources afterward.

From the notebooks

Handling large-scale embedding generation for Agent Platform Vector Search

  • Throttle API calls to stay within embedding quota limits.
  • Use multithreading to reduce latency impact and improve quota utilization.
  • Checkpoint generated embeddings periodically to Cloud Storage.
  • Record errors to a log file during long-running jobs.
  • Use moderate text batch sizes such as 20 instead of always using the maximum 250.

Intro to Gemini Agentic Vision

  • Enable code execution when the model needs to crop, inspect, calculate, plot, or draw instead of guessing from a static image.
  • Parse response parts explicitly to inspect reasoning text, generated code, execution output, and resulting images.
  • Use PIL and IPython display to render images returned from code execution.
  • Use structured image parts from bytes or URI with the correct MIME type.

Evaluate images with Gecko

  • Generate rubrics from each prompt before validation so the metric adapts to prompt-specific challenges.
  • Use CustomOutputConfig with return_raw_output=True and parsing_fn for custom validator outputs.
  • Compare matching prompts with similar counterexample prompts to show high-quality and low-quality responses.
  • Inspect generated questions and validator reliability, and manually add questions when needed.
  • Aggregate validation results into final scores and per-question QA results.

Evaluating multimodal task

  • Use a structured response schema with score and explanation for autorater outputs.
  • Define a clear metric definition, criteria, rating rubric, and evaluation steps in the custom metric prompt.
  • Include both summary metrics and row-based metrics when reviewing evaluation results.
  • Inspect sampled explanations to validate the evaluation behavior.
  • Clean up the Vertex AI experiment when it is no longer needed.

Evaluate videos with Gecko

  • Generate rubrics from the prompt so metrics reflect prompt-specific challenges.
  • Separate rubric generation from the validator step.
  • Use custom parsing functions for rubric generation and validation outputs.
  • Inspect generated questions and validator reliability before relying on the score.
  • Include matching and counterexample prompts to demonstrate score differences.

Evaluating prompts at scale with Gemini Batch Prediction API

  • Test the prompt on one image with generate_content before launching batch prediction.
  • Request JSON output in both the prompt and generation config.
  • Store evaluation_ts, evaluation_id, prompt_text, and gcs_uri with every request row.
  • Load ground truth into BigQuery and join predictions to compute correctness.
  • Use BigQuery views to parse raw responses and expose an evaluation table.

Rubric evaluation - Multimodal and Custom metric for text quality

  • Inspect, edit, or add generated rubrics before final evaluation.
  • Use predefined rubric metrics when they fit the multimodal task.
  • Use custom prompt templates when text-quality criteria must be controlled.
  • Return raw autorater output to debug reasoning and verdicts.
  • Write rubrics as granular binary yes/no constraints and avoid hallucinated or repeated criteria.

Evaluate Gemini Structured Output

  • Use structured output with response_mime_type application/json and response_schema for consistent JSON.
  • Keep reference ground truth alongside each model response in the evaluation dataset.
  • Evaluate both schema validity and extraction accuracy with custom metrics.
  • Use DeepDiff to inspect field-wise differences between reference and response.
  • Compare multiple Gemini model ids over the same prompt and input images.

Evaluate images with predefined Gecko

  • Generate rubrics from the user prompts before evaluating responses.
  • Use counterexample prompts with the same images to demonstrate high and low quality evaluations.
  • Inspect generated questions and validator reliability when analyzing quality.
  • Manually add questions when desired for an application.
  • Review Vertex AI pricing and estimate costs before running evaluation.

Nano Banana 🍌: Gemini 2.5 Flash Image Recipes

  • Defines shared model and generation configuration before running recipes.
  • Uses helper functions to create canvases and display generated images consistently.
  • Checks response candidates and inline image data before displaying output.
  • Uses blank canvases to guide aspect ratio-sensitive outputs.
  • Uses source and reference images with text prompts for grounded image edits.

Get Started with Vertex AI Prompt Optimizer - Multimodality

  • Use labeled rows containing query, image GCS URI, and target for question_answering_correctness evaluation.
  • Use examples where the current system instruction performs poorly when building an optimization dataset.
  • Validate optimizer settings with the Pydantic OptimizationConfig before submitting the job.
  • Store both optimizer configuration and results in Cloud Storage.
  • Set has_multimodal_inputs to True when optimizing prompts with image inputs.

Supervised Fine Tuning with Gemini 2.5 Flash for Image Captioning

  • Use high-quality, well-labeled, task-relevant training data because low-quality data can hurt performance and introduce bias.
  • Use a separate validation set to evaluate model performance.
  • Choose evaluation metrics that reflect the task; this notebook uses ROUGE for image caption text generation.
  • Experiment with generation parameters and prompt structures to improve task performance.
  • Start with recommended default tuning hyperparameters, then customize epochs, learning rate multiplier, or adapter size for specific needs.

Supervised Fine Tuning with Gemini 2.0 Flash for change detection using the Google Gen AI SDK

  • Use JSON Lines format with one tuning example per line for Gemini supervised fine-tuning.
  • Copy tutorial data into your own Cloud Storage bucket before preparing tuning files.
  • Use default tuning settings for initial runs because the notebook says they are recommended for optimal performance.
  • Set epoch_count=1 in the tutorial to keep time and cost low.
  • Use temperature 0 when comparing a tuned model response with ground truth qualitatively.

Supervised Fine-tuning Gemini 2.5 Flash for Visual Defect Detection

  • Uses a separate train, validation, and test split with an 80/10/10 split.
  • Stores image files and JSONL tuning data in Cloud Storage for the fine-tuning service.
  • Uses guard clauses for missing data, unavailable endpoints, unexpected sample formats, and response parsing.
  • Sets a low temperature for deterministic classification during evaluation.
  • Uses a smaller adapter size for faster tuning in the demo.

Semantic Analysis in BigQuery with AI Functions

  • Use managed AI functions for analysts who want prompt optimization handled by BigQuery.
  • Use AI.SCORE for semantic ranking by subjective criteria.
  • Use AI.CLASSIFY with explicit categories and a fallback category such as All Pets.
  • Use AI.IF for semantic filtering in WHERE clauses and semantic joins in JOIN ON clauses.
  • Use AI.GENERATE_BOOL for SELECT-clause enrichment when full prompt control is needed.

Text + multimodal embedding generation and vector search in BigQuery

  • Create a dedicated BigQuery dataset before loading tutorial tables.
  • Use BigQuery remote models instead of exporting data to call Vertex AI separately.
  • Store embeddings in ARRAY columns for reuse with VECTOR_SEARCH.
  • Combine product_name and description before generating text embeddings to improve semantic context.
  • Order vector search results by distance ascending.

Document Processing with Gemini

  • Use explicit system instructions for extraction, classification, QA, and summarization tasks.
  • Use response_schema and response_mime_type to enforce structured JSON or enum outputs.
  • Use Pydantic models to define extraction schemas with field descriptions.
  • Use Part.from_uri for GCS or HTTPS PDFs and Part.from_bytes for local PDF bytes.
  • Use response.parsed when consuming structured model outputs.

Patents Document Understanding with Gemini

  • Use a detailed prompt plus responseMimeType application/json and responseSchema for controlled generation.
  • Include the PDF as fileData with mimeType application/pdf instead of extracting document text separately.
  • Use a systemInstruction to set the model role as an expert at analyzing patent documents.
  • Write batch requests to BigQuery and let the batch job return outputs to a BigQuery table.
  • Poll batch job state before reading destination results.

Automating Income Taxes with Gemini

  • Use temperature=0 for deterministic classification and extraction.
  • Constrain classification with an Enum response schema.
  • Use document-specific Pydantic schemas for typed structured extraction.
  • Map classified document types to their extraction schemas before calling the model.
  • Normalize parsed JSON before joining it back into the source DataFrame.

Using Gemini in Education

  • Use a low default temperature for more consistent responses.
  • Use few-shot examples to guide response structure and formatting for text correction.
  • Ask for step-by-step reasoning to reduce hallucinations in math tasks.
  • Use structured output requests such as JSON lists or tables when asking detailed questions.
  • Ask video questions using the video only and request timestamps plus source type such as image, text, or speech.

Gemini: An Overview of Multimodal Use Cases

  • Pass non-text inputs with explicit MIME types using Part.from_uri.
  • Use context caching for repeated questions over a large codebase instead of resending the same prompt.
  • Set temperature to 0 for factual identification prompts, as shown for the train-line example.
  • Label candidate images in the prompt when asking the model to choose among provided images.
  • Tell the model not to make up information when answers must be grounded only in attached audio or video.

Creative Content Generation with Gemini in Vertex AI and Imagen

  • Add product description to improve generated output quality.
  • Use product images as multimodal context for marketing messages.
  • Target generated posts to specific platforms such as Facebook, Instagram, LinkedIn, and Twitter.
  • Reuse generated posts as reference context when personalizing for audience segments.
  • Prepare an expanded base image and mask before calling Imagen outpainting.

Generating Consistent Imagery with Gemini 🍌

  • Use environment variables or Colab Secrets instead of hardcoding API configuration.
  • Use a character sheet as a reusable design reference for future image-generation tasks.
  • Refer explicitly to input images, such as Image 1 and Image 2, to avoid ambiguity.
  • Clarify removed or changed objects, such as no longer holding the map or removing ice axes.
  • Spend time refining the first scene because it cascades into later generated scenes.

Video Captioning with Gemini

  • Use a detailed system prompt that specifies perspective, camera movement, subject motion, setting, lighting, body language, and visible text.
  • Return only the caption by instructing the model not to send a preamble.
  • Count video tokens separately when token accounting matters.
  • Return usage metadata, model version, generated text, and video-only token count for traceability.
  • Review the sample video before captioning to understand visual, textual, and audio elements.

Video Data Curation - Video Quality Filtering

  • Discard low-quality clips to use limited modeling compute efficiently.
  • Use metadata filters such as FPS, duration, resolution, brightness, and aspect ratio for downstream modeling requirements.
  • Use structured JSON output for separating detected text from watermark descriptions.
  • Set Gemini temperature to 0.0 for deterministic extraction-style calls.
  • Manually inspect low-scoring videos to choose an aesthetic score threshold.

Data Curation Pipeline: Splitting and Transcoding

  • Store videos in Cloud Storage before calling the Multimodal Embeddings API.
  • Use a semaphore with ThreadPoolExecutor to control embedding API concurrency.
  • Capture errors per video URI instead of failing the whole embedding run.
  • Store embeddings in BigQuery for reproducible vector search workflows.
  • Create a BigQuery VECTOR INDEX with COSINE distance before large vector search workloads.

Data Curation Pipeline: Splitting and Transcoding

  • Filter videos by duration and resolution before downstream processing.
  • Use FFmpeg directly via subprocess for robust fixed-length splitting with stream copy.
  • Clean up temporary files and directories after local video processing.
  • Sort segment files before batch transcoding for consistent processing order.
  • Test transcoding parameters on a sample of video data before applying them broadly.

Productivity Coaching with Gemini and Google Calendar

  • Use system instructions to define Gemini’s coaching role and analysis criteria.
  • Start with sample calendar screenshots before connecting live user data.
  • Request only readonly Calendar API access for event analysis.
  • Validate tool input bounds before calling the Calendar API.
  • Return a concise subset of event fields: summary, start, end, and status.

Product attributes extraction and detailed descriptions from images using Gemini 2.0

  • Use temperature 0 for deterministic attribute extraction.
  • Provide system instructions that constrain answers to visible product evidence.
  • Use a closed vocabulary when attribute values must come from an approved set.
  • Parse model JSON output before returning it to application code.
  • Use debug mode to display the image and prompt during development.

Building a Multimodal Chatbot for Warranty Claims using Gemini and Vector Search in Vertex AI

  • Use a unique lowercase RAG identifier without spaces for generated resources.
  • Keep chunk overlap when splitting retrieved text to preserve context across chunks.
  • Store Vector Search input as JSONL with id and embedding fields.
  • Use retrieved context in the prompt and instruct Gemini to answer only from provided text.
  • Return a fallback response when no matching page source is found.

Detecting and Editing Visual Objects with Gemini

  • Store environment configuration outside source code with environment variables, Colab Secrets, or platform defaults.
  • Use structured outputs with response_mime_type and response_schema for automated parsing.
  • Keep bounding box granularity controlled through prompt wording.
  • Use precise prompts such as excluding captions from boxes or preserving original line breaks.
  • Use low randomness, seed, and image response modalities for more deterministic restoration outputs.

Spatial understanding with Gemini 3

  • Use response_mime_type=“application/json” and response_schema=list[BoundingBox] for controlled generation.
  • Put repeated output-format rules in system_instruction to keep prompts shorter.
  • Use safety_settings with HARM_CATEGORY_DANGEROUS_CONTENT set to BLOCK_ONLY_HIGH.
  • Give repeated objects unique labels based on distinct characteristics such as color, size, or position.
  • Scale normalized box coordinates by image width and height before drawing overlays.

Unlocking Multimodal Video Transcription with Gemini

  • Use environment variables or Colab Secrets instead of hardcoding API configuration.
  • Start with simple prompts to observe Gemini’s natural behavior before refining instructions.
  • Craft prompts iteratively, precisely, and concisely.
  • Decouple transcripts and speakers into linked tables with a consistent voice ID.
  • Start transcript generation with audio-focused work before extracting speaker names from visual and audio cues.

Identifcation of Scene Transitions in Movies Using Gemini

  • Use temperature 0 for consistent scene boundary extraction.
  • Provide both video and VTT inputs to combine visual and dialogue cues.
  • Use response_mime_type application/json with response_schema for structured outputs.
  • Define explicit criteria for narrative, visual, dialogue, audio, and cohesion signals.
  • Do not count jump-cuts or insert shots as transitions unless they indicate meaningful narrative shifts.

YouTube Video Analysis with Gemini

  • Use Part.from_uri with mime_type=“video/webm” for public YouTube video inputs.
  • Use response_mime_type=“application/json” and response_schema for structured extraction.
  • Constrain enum fields in the response schema when only specific values are valid.
  • Prompt the model to use only information in the video itself for extraction.
  • Use asynchronous generation for analyzing multiple videos more efficiently.

Enhanced Vision Assistant with Gemini

  • Use environment variables for GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION when possible.
  • Allow an explicit credentials path but also support application default credentials.
  • Filter small detections with MIN_OBJECT_SIZE before generating guidance.
  • Prioritize navigation guidance by urgency using NavigationPriority.
  • Clean up camera/audio resources and temporary audio files in stop().

Serving Gemma 3 with vLLM on Cloud Run

  • Use Secret Manager to pass the Hugging Face token to Cloud Build instead of baking it into code.
  • Download model weights at image build time and set HF_HUB_OFFLINE=1 for cached runtime loading.
  • Keep the Cloud Run service private and rely on IAM authentication.
  • Set max instances according to available NVIDIA L4 GPU quota.
  • Use Artifact Registry for the built serving image.

Hugging Face DLCs: Serving PaliGemma using Pytorch Inference on Vertex AI with Custom Handler

  • Test the handler logic locally before packaging it as handler.py.
  • Use HF_HOME to control the local Hugging Face cache location.
  • Upload large model files to Cloud Storage with gsutil parallel composite uploads.
  • Validate request instances contain both prompt and image_base64.
  • Use torch.inference_mode() for generation.

Multimodal Function Calling with Claude Models

  • Select a Claude model before creating the AnthropicVertex client because models have different location availability.
  • Use environment variables GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION when explicit project or region values are not provided.
  • Define tools with a name, description, and input_schema.
  • Append the assistant tool-use response and a user tool_result message before asking Claude for the final answer.
  • Pass tools again in the follow-up messages.create call.

Imagen 3 Image Editing

  • Set safety_filter_level and person_generation in generation and editing configs.
  • Use RawReferenceImage for the source image and MaskReferenceImage for mask-based edits.
  • Use MASK_MODE_USER_PROVIDED when supplying your own mask image.
  • Use an empty prompt for inpainting removal requests where the object should simply be removed.
  • Display original and edited images side by side for visual comparison.

Imagen 4 Image Upscale

  • Install or upgrade google-genai before using the notebook.
  • Authenticate only in Colab by checking for google.colab in sys.modules.
  • Keep generation and upscaling model IDs in variables.
  • Use types.GenerateImagesConfig for aspect ratio, image count, and image size.
  • Use types.Image.from_file for local files and types.Image(gcs_uri=…) for Cloud Storage images.

Veo 3.1 Advanced Controls

  • Use Cloud Storage URIs directly for remote images instead of downloading when appropriate.
  • Poll long-running operations with client.operations.get before reading results.
  • Set output_gcs_uri for larger generated videos and video extension outputs.
  • Specify mime_type when passing Cloud Storage image or video URIs.
  • Use SynthID-watermarked Veo outputs as noted by the notebook.

Veo 3.1 Reference to Video

  • Use asset reference images for subjects, objects, or scenes that should appear in the final video.
  • Set aspect_ratio, number_of_videos, duration_seconds, resolution, person_generation, and generate_audio explicitly.
  • Poll the operation before reading generated_videos.
  • Use Cloud Storage URIs for multiple reference images when avoiding local downloads.
  • All Veo videos include SynthID digital watermarking.

Virtual Try-On: Image Generation

  • Use Image.from_file for local person and product images.
  • Save an intermediate try-on output locally when it will be reused in a later request.
  • Set output_mime_type and number_of_images explicitly in RecontextImageConfig or GenerateImagesConfig.
  • Use safety_filter_level explicitly when generating or recontextualizing images.
  • Combine multiple clothing items into one photo when trying on multiple items in a single request.

Virtual Try-On: Batch Generation Pipeline

  • Reads project and region from environment variables when notebook parameters are not set.
  • Displays input images before generation and displays generated images afterward for visual inspection.
  • Uses response.raise_for_status() when fetching public image URLs.
  • Tracks current job and total jobs while processing the Cartesian product of inputs.
  • Catches per-job exceptions and skips failed jobs instead of stopping the whole batch.

Back to Vision · Best Practices Map

Graph View

Backlinks