Vision

Vision on Google Cloud covers multimodal understanding, generation, editing, embedding, evaluation, and serving workflows for images, video, PDFs, and visual documents using Gemini, Imagen, Veo, Vertex AI, Agent Platform, BigQuery, Cloud Storage, and related SDKs.

Synthesized across 45 notebooks. See the source pages under Notebooks below for line-level grounding.

The Vision notebooks span practical engineering patterns for visual media: asking Gemini questions about images, videos, documents, diagrams, products, calendars, and screenshots; extracting structured fields; generating captions; detecting objects; and grounding answers in provided media.

They also cover production-adjacent workflows: large-scale embedding generation, Vector Search, BigQuery AI functions, batch prediction, prompt optimization, supervised fine-tuning, evaluation with rubrics and autoraters, and deployment of open or partner multimodal models.

The recurring theme is to treat visual AI as a data pipeline, not a single prompt: configure project access, store media reliably, pass explicit MIME-typed inputs, constrain outputs with schemas, evaluate against references, throttle and checkpoint at scale, and clean up billable resources.

Canonical workflow

1. Enable the required APIs, billing, authentication, project ID, region, and service-account permissions for Vertex AI, Agent Platform, BigQuery, Cloud Storage, or companion APIs.
2. Stage images, videos, PDFs, captions, labels, and reference data in Cloud Storage or BigQuery, and pass media with explicit MIME types using the SDK-supported URI, bytes, or image object formats.
3. Choose the right model or service for the task: Gemini for multimodal understanding, Imagen or Gemini image models for image generation and editing, Veo for video generation, embeddings for retrieval or deduplication, and BigQuery AI functions for in-database workflows.
4. Prototype on one or a few examples before scaling, inspecting the input media, prompt, response parts, parsed output, usage metadata, and visual results.
5. Use structured output contracts for automation, including JSON response MIME types, response schemas, Pydantic models, enums, closed vocabularies, and parsing functions.
6. Scale with Batch Prediction, BigQuery, Vector Search, pipelines, Dataflow, multithreading, semaphores, throttling, checkpointing, and per-item error logging.
7. Evaluate outputs with ground truth, custom metrics, generated rubrics, counterexamples, row-level explanations, validation questions, and manual spot checks.
8. Optimize, tune, deploy, and clean up: use Prompt Optimizer or supervised fine-tuning when labeled data justifies it, poll long-running jobs, secure serving endpoints, and delete unused endpoints, buckets, experiments, indexes, and datasets.

Best practices

See Vision - Best Practices for the full distilled guide.

• Pass every non-text input with the correct MIME type, using supported structures such as Part.from_uri, Part.from_bytes, fileData, Image.from_file, gcs_uri images, or BigQuery ObjectRefs.
• Use response_mime_type=“application/json”, response_schema, Pydantic models, enums, and custom parsers whenever downstream code depends on model output.
• Set temperature to 0 or a low value for deterministic extraction, classification, scene-boundary detection, attribute labeling, and evaluation comparisons.
• Ground prompts explicitly: tell the model to use only the provided media or retrieved context, label candidate images, constrain answers to visible evidence, and request no preamble when only the answer is needed.
• Prototype with a single image, video, document, or prompt before launching batch prediction, tuning, vector indexing, or large generation jobs.
• Inspect intermediate artifacts: response parts, generated code execution output, parsed JSON, rubrics, validator questions, row-level explanations, visual overlays, captions, and side-by-side edited images.

Key models & APIs

• Gemini multimodal models: gemini-2.0-flash, gemini-2.0-flash-001, gemini-2.5-flash, gemini-2.5-pro, gemini-3.5-flash, gemini-3.1-pro-preview, gemini-3.1-flash-lite, Gemini image models: gemini-2.5-flash-image-preview, gemini-2.5-flash-image, gemini-3-pro-image, gemini-3.1-flash-image, Imagen models: imagen-3.0-generate-002, imagen-3.0-capability-001, imagen-4.0-generate-001, imagen-4.0-upscale-preview, imagegeneration@006, Veo models: veo-3.1-generate-001, veo-3.1-generate-preview, veo-3.1-fast-generate-preview, Virtual Try-On: virtual-try-on-001, Embedding models: gemini-embedding-001, gemini-embedding-2-preview, multimodalembedding@001, text-embedding-005, google/siglip2-base-patch16-512, Open and partner models on Google Cloud: google/paligemma-3b-mix-448, google/gemma-3-4b-it, claude-3-5-sonnet-v2@20241022, Vertex AI API, Agent Platform API, Google Gen AI SDK, google-cloud-aiplatform, and google-genai, Vertex AI Gen AI Evaluation, EvalTask, Batch Prediction API, Prompt Optimizer, supervised fine-tuning, and Vertex AI endpoints, BigQuery AI functions, BigQuery remote models, BigQuery ObjectRefs, BigQuery VECTOR_SEARCH, BigQuery vector indexes, and Agent Platform Vector Search, Cloud Storage, Cloud Vision API, Cloud Text-to-Speech API, Google Calendar API, Cloud Run GPU, Artifact Registry, Secret Manager, and PyTorch inference DLCs

Notebooks (45)

Embeddings & Vector Search (1)

• Handling large-scale embedding generation for Agent Platform Vector Search — Generates large-scale text and image embeddings for Agent Platform Vector Search with throttling and checkpointing.

Gemini (35)

• Intro to Gemini Agentic Vision — Uses Gemini code execution for agentic vision tasks: zooming, plotting, and image annotation.
• Evaluate images with Gecko — Evaluates prompt-image alignment with Gecko-style rubric generation and VQA validation in Vertex AI.
• Evaluating multimodal task — Evaluates image-grounded car damage labels with Vertex AI EvalTask and a Gemini custom autorater.
• Evaluate videos with Gecko — Uses Vertex AI evaluation to run Gecko-style rubric generation and video validation.
• Evaluating prompts at scale with Gemini Batch Prediction API — Evaluates Gemini image-classification prompts at scale with Batch Prediction and BigQuery.
• Rubric evaluation - Multimodal and Custom metric for text quality — Evaluates multimodal car-damage responses and text summaries with Vertex AI rubric-based metrics
• Evaluate Gemini Structured Output — Evaluates Gemini structured JSON extraction from scanned order forms with Vertex AI Gen AI Evaluation.
• Evaluate images with predefined Gecko — Evaluates text-to-image outputs with Vertex AI predefined Gecko image rubrics.
• Nano Banana 🍌: Gemini 2.5 Flash Image Recipes — Demonstrates Gemini 2.5 Flash image generation and editing recipes on Vertex AI with google-genai.
• Get Started with Vertex AI Prompt Optimizer - Multimodality — Optimizes a multimodal Gemini prompt with Vertex AI Prompt Optimizer on MathVista image QA.
• Supervised Fine Tuning with Gemini 2.5 Flash for Image Captioning — Fine-tunes Gemini 2.5 Flash on GCS image-caption pairs and evaluates ROUGE before and after tuning.
• Supervised Fine Tuning with Gemini 2.0 Flash for change detection using the Google Gen AI SDK — Fine-tunes Gemini 2.0 Flash on paired images for spot-the-difference change detection.
• Supervised Fine-tuning Gemini 2.5 Flash for Visual Defect Detection — Fine-tunes Gemini 2.5 Flash on image-based manufacturing defect classification.
• Semantic Analysis in BigQuery with AI Functions — Uses BigQuery AI functions with Gemini to rank, classify, filter, join, and enrich pet product data.
• Text + multimodal embedding generation and vector search in BigQuery — Builds text and image embeddings in BigQuery for semantic product search.
• Document Processing with Gemini — Processes PDFs with Gemini on Vertex AI for extraction, classification, QA, summarization, translation, and page selection.
• Patents Document Understanding with Gemini — Uses Gemini batch prediction on Vertex AI to extract structured fields and figure boxes from patent PDFs.
• Automating Income Taxes with Gemini — Classifies tax PDFs and extracts structured tax fields with Gemini on Vertex AI.
• Using Gemini in Education — Demonstrates Gemini education prompts across text, math, images, multiple images, and video.
• Gemini: An Overview of Multimodal Use Cases — Shows Gemini multimodal prompts for text, PDFs, images, video, audio, code, retail, diagrams, and comparisons.
• Creative Content Generation with Gemini in Vertex AI and Imagen — Generates and personalizes GShoe marketing copy with Gemini, then outpaints product images with Imagen.
• Generating Consistent Imagery with Gemini 🍌 — Generates a consistent robot image story from one archive image using Gemini 2.5 Flash Image.
• Video Captioning with Gemini — Uses Gemini on Vertex AI to generate rich multimodal captions from a video in Cloud Storage.
• Video Data Curation - Video Quality Filtering — Filters video datasets by metadata, OCR/watermarks, aesthetics, and motion scores for curation.
• Data Curation Pipeline: Splitting and Transcoding — Deduplicates video clips using video embeddings and BigQuery vector search.
• Data Curation Pipeline: Splitting and Transcoding — Builds a video curation pipeline for metadata filtering, splitting, scene detection, and transcoding.
• Productivity Coaching with Gemini and Google Calendar — Uses Gemini 2.5 Flash with Google Calendar data to provide productivity coaching and schedule recommendations.
• Product attributes extraction and detailed descriptions from images using Gemini 2.0 — Extract product attributes and detailed retail descriptions from images with Gemini 2.0 on Vertex AI.
• Building a Multimodal Chatbot for Warranty Claims using Gemini and Vector Search in Vertex AI — Builds a multimodal warranty-claims chatbot with Gemini, RAG, Vector Search, and function calling.
• Detecting and Editing Visual Objects with Gemini — Detects visual objects with Gemini, crops them, and edits them with Gemini image models.
• Spatial understanding with Gemini 3 — Uses Gemini 2.5 Flash on Vertex AI to detect objects and draw normalized bounding boxes on images.
• Unlocking Multimodal Video Transcription with Gemini — Uses Gemini to build multimodal video transcripts with voice IDs, speaker names, and structured JSON output.
• Identifcation of Scene Transitions in Movies Using Gemini — Uses Gemini to detect movie scene transitions from a video and matching VTT subtitles.
• YouTube Video Analysis with Gemini — Analyzes public YouTube videos with Gemini for summaries, structured JSON extraction, and cross-video insights.
• Enhanced Vision Assistant with Gemini — Builds a Gemini vision assistant that detects objects, assesses hazards, and speaks navigation guidance.

Open Models (2)

• Serving Gemma 3 with vLLM on Cloud Run — Deploys Gemma 3 on Cloud Run with vLLM as an OpenAI-compatible chat API.
• Hugging Face DLCs: Serving PaliGemma using Pytorch Inference on Vertex AI with Custom Handler — Deploys gated PaliGemma from Hugging Face to Vertex AI using a PyTorch DLC and custom handler.

Partner Models (1)

• Multimodal Function Calling with Claude Models — Uses Claude on Vertex AI for multimodal prompting and tool use with a Wikipedia lookup.

Vision (6)

• Imagen 3 Image Editing — Edits images with Imagen 3 using inpainting, background swap, outpainting, and mask-free prompts.
• Imagen 4 Image Upscale — Shows how to upscale generated, local, and Cloud Storage images with Imagen 4 using google-genai.
• Veo 3.1 Advanced Controls — Shows Veo 3.1 frame interpolation and video extension with Google Gen AI SDK on Agent Platform.
• Veo 3.1 Reference to Video — Generates Veo 3.1 videos from reference images using the Google Gen AI SDK.
• Virtual Try-On: Image Generation — Uses Google Gen AI SDK to generate virtual try-on images from local and Cloud Storage clothing images.
• Virtual Try-On: Batch Generation Pipeline — Batch-generates virtual try-on images from person and apparel inputs using Google GenAI SDK.

Related concepts

• Gemini Capabilities · Applied Use Cases · Image & Video Generation · Evaluation · Prompt Engineering · Function Calling & Tools · Getting Started · Embeddings & Vector Search

---

Part of the Generative AI Google Brain · Vision - Best Practices