Gemini Capabilities

Gemini Capabilities on Google Cloud are the model features, APIs, and platform patterns used to build production generative AI systems with Gemini across text, multimodal input, structured output, tools, memory, grounding, evaluation, batch inference, embeddings, speech, and agents. On Google Cloud, these capabilities are primarily accessed through Vertex AI, Agent Platform, Google Gen AI SDK, REST, BigQuery, Cloud Storage, and related managed services.

Synthesized across 81 notebooks. See the source pages under Notebooks below for line-level grounding.

This area covers how Gemini models are used beyond basic text generation: controlled JSON output, multimodal reasoning over images/audio/video/PDFs, function calling, code execution, grounding, long context, context caching, embeddings, batch prediction, speech generation, agent memory, computer use, and evaluation.

The common engineering pattern is to make the model interaction explicit: configure the project, location, model, tools, safety settings, schemas, media parts, and generation parameters per request; inspect response metadata; and route model outputs into application code, BigQuery, Cloud Storage, Vertex AI Pipelines, Text-to-Speech, or evaluation workflows.

Most notebooks emphasize production constraints: authentication, API enablement, regional availability, quota, token cost, safety blocks, model-specific feature support, and cleanup of billable resources. Reliable systems preserve chat/tool history correctly, validate structured outputs, ground responses when facts matter, and evaluate quality with task-specific metrics.

Canonical workflow

1. Set up a Google Cloud project, enable the required APIs, authenticate with Application Default Credentials, and configure PROJECT_ID, LOCATION, and any service-specific environment variables.
2. Choose the appropriate Gemini model and access path for the task, such as Google Gen AI SDK, Vertex AI REST, OpenAI-compatible Chat Completions, Agent Platform, BigQuery ML, or Text-to-Speech APIs.
3. Prepare inputs deliberately: add system instructions, prompt context, examples, schemas, media parts with correct MIME types, tool declarations, memory scope, grounding source, or batch input tables/files.
4. Configure generation behavior explicitly, including thinking controls, safety settings, response modalities, response schemas, function-calling mode, media resolution, context caching, or task-specific embedding type.
5. Run the request or job, using streaming, async, batch prediction, Vertex AI Pipelines, or BigQuery jobs when those execution modes fit latency and scale requirements.
6. Inspect outputs and metadata, including candidates, parts, function calls, code execution results, grounding metadata, usage metadata, cached token counts, safety ratings, finish reasons, and job status.
7. Post-process safely in application code: validate JSON, execute trusted tools, return tool responses, synthesize audio, store embeddings, query vector indexes, sample BigQuery outputs, or persist memories.
8. Evaluate, monitor, and clean up by comparing against references, running Vertex AI Evaluation or custom metrics, polling terminal job states, deleting caches/jobs/agents/buckets/datasets when done, and preserving IAM policies before updates.

Best practices

See Gemini Capabilities - Best Practices for the full distilled guide.

• Keep model configuration explicit per request with GenerateContentConfig, system_instruction, safety_settings, tools, response_schema, response_mime_type, generation parameters, and model-selection or thinking controls.
• Use structured output for downstream systems by setting response_mime_type to application/json, defining response_schema or response_json_schema, marking required fields, allowing nullable fields when context may be missing, and validating parsed results.
• Use low temperature or temperature=0 for deterministic extraction, classification, tool-calling, and code-execution examples; keep Gemini 3+ sampling defaults when the notebook recommends default temperature behavior.
• Preserve full model response content in manual function-calling or tool histories so thought signatures are retained; prefer SDK chat history or automatic function calling when possible.
• Declare tools with clear FunctionDeclaration names, descriptions, typed parameters, required fields, and allowed function modes; execute tools only in trusted application code and verify function call names before dispatch.
• Use grounding when factuality matters: inspect grounding metadata, citations, search queries, retrieval queries, and grounding chunks, and choose Google Search, Enterprise Web Search, Maps, Vertex AI Search, or internal context based on compliance and data needs.

Key models & APIs

• gemini-3.5-flash, gemini-3.1-pro-preview, gemini-3.1-flash-lite, gemini-3-flash-preview, gemini-2.5-pro, gemini-2.5-flash, gemini-2.5-flash-lite, gemini-2.5-flash-image, gemini-2.0-flash, gemini-2.0-flash-001, gemini-3.1-flash-tts-preview, gemini-2.5-flash-tts, gemini-embedding-001, gemini-embedding-2, text-embedding-005, gemini-2.5-computer-use-preview-10-2025, model-optimizer-exp-04-09, Google Gen AI SDK, Vertex AI Gemini API, Vertex AI REST API, OpenAI Chat Completions-compatible Vertex AI endpoint, Agent Platform API, Vertex AI Agent Engine, Vertex AI Memory Bank, ADK, Gemini Data Analytics API, Cloud Text-to-Speech API, Vertex AI Evaluation, Vertex AI Pipelines, BigQuery ML ML.GENERATE_TEXT, BigQuery ML ML.GENERATE_EMBEDDING, BigQuery Vector Search, Vertex AI Search, Google Search grounding, Enterprise Web Search grounding, Google Maps grounding, Cloud Storage, BigQuery, LangGraph, LangChain, Playwright

Notebooks (81)

Agents & ADK (2)

• Building a Multimodal Trip Planner with ADK on Vertex AI Agent Engine Memory Bank — Builds and deploys a multimodal ADK trip planner using Vertex AI Agent Engine Memory Bank.
• Intro to Gemini Data Analytics — Shows REST-based Gemini Data Analytics agents over BigQuery, Looker, or Looker Studio data.

Audio (3)

• Get started with Gemini-TTS voices using Text-to-Speech — Synthesizes Gemini-TTS speech with Cloud Text-to-Speech and Agent Platform APIs.
• Create a Multi-Speaker Podcast with Gemini 2.0 & Text-to-Speech — Generates a two-speaker podcast from a PDF using Gemini JSON output and Text-to-Speech MP3 synthesis.
• Narrate a Multi-character Story with Gemini and Text-to-Speech — Generates a Gemini play script and narrates it with distinct Text-to-Speech voices per character.

Embeddings & Vector Search (2)

• Introduction to Gemini Multimodal Embeddings — Generates Gemini multimodal embeddings and applies them to search, similarity, PDFs, and RAG.
• Anomaly Detection of Infrastructure Logs using Gemini and BigQuery Vector Search — Detects HDFS log anomalies with Gemini summaries, text embeddings, and BigQuery Vector Search.

Gemini (72)

• Get started with Vertex AI Memory Bank - ADK — Builds an ADK agent with Vertex AI Memory Bank for long-term user memory across sessions.
• Semantic Router Agent — Builds a LangGraph semantic router that sends retail queries to mocked Gemini expert assistants.
• Intro to Batch Inference with the Gemini API — Runs Gemini batch inference jobs using Cloud Storage and BigQuery inputs and outputs.
• Monitor batch prediction with Gemini API — Orchestrates and monitors Gemini batch predictions with Vertex AI Pipelines and BigQuery output.
• Using OpenAI libraries with Gemini on Vertex AI — Calls Gemini on Vertex AI through OpenAI Chat Completions with streaming, tools, schemas, caching, and safety.
• Intro to Gemini Agentic Vision — Uses Gemini code execution for agentic vision tasks: zooming, plotting, and image annotation.
• Intro to Generating and Executing Python Code with Gemini 3 — Uses Gemini 3.5 Flash code execution to generate, run, inspect, and stream Python results via Vertex AI.
• Intro to Computer Use with Gemini — Builds a Gemini Computer Use browser agent loop with Playwright and screenshots.
• Intro to Context Caching with the Gemini API — Shows implicit and explicit context caching for Gemini in Vertex AI with the Google Gen AI SDK.
• Intro to Structured Output with the Gemini API — Shows how to make Gemini 2.5 Flash return JSON, enums, and schema-constrained multimodal outputs.
• Enhancing quality and explainability with Vertex AI Evaluation — Ranks Gemini answers with Vertex AI pairwise and pointwise evaluation explanations.
• Evaluate Gemini Structured Output — Evaluates Gemini structured JSON extraction from scanned order forms with Vertex AI Gen AI Evaluation.
• Forced Function Calling with Tool Configurations in Gemini — Shows how to force, allow, or disable Gemini function calls with tool configurations.
• Multimodal Function Calling with the Gemini API & Python SDK — Shows Gemini multimodal function calling with images, video, audio, PDFs, and chat using Google Gen AI SDK.
• Working with Parallel Function Calls and Multiple Function Responses in Gemini — Shows how Gemini returns and handles parallel function calls with Wikipedia lookup tools.
• Intro to Gemini 2.5 Flash — Introduces Gemini 2.5 Flash on Vertex AI with text, thinking, multimodal, tools, and structured output.
• Intro to Gemini 2.5 Flash-Lite — Introduces Gemini 2.5 Flash-Lite on Vertex AI with prompting, thinking, structured output, search, and code tools.
• Gemini 2.5 Flash Image (Nano Banana 🍌) Generation — Generates and edits images with Gemini 2.5 Flash Image using the Google Gen AI SDK on Agent Platform.
• Intro to Gemini 2.5 Pro — Introduces Gemini 2.5 Pro on Vertex AI using the Google Gen AI SDK for text, multimodal, tools, and reasoning.
• Intro to Gemini 3.1 Flash-Lite — Quickstart for Gemini 3.1 Flash-Lite with Google Gen AI SDK on Google Cloud.
• Intro to Gemini 3.1 Pro — Quickstart for Gemini 3.1 Pro on Vertex AI with Google Gen AI SDK API features.
• Intro to Gemini 3.5 Flash — Quickstart for Gemini 3.5 Flash generation, multimodal inputs, tools, grounding, and safety with Google GenAI SDK.
• Intro to Gemini 3 Flash (Preview) — Quickstart for Gemini 3 Flash Preview on Vertex AI with Gen AI SDK features and multimodal/tool examples.
• Getting Started with Chat with Gemini — Shows stateful Gemini chat with Google GenAI SDK and LangChain on Vertex AI.
• REST API — Uses cURL to call Gemini 3.5 Flash on Vertex AI for text, chat, tools, multimodal, and JSON output.
• Getting started with Gemini using Vertex AI in Express Mode — Introduces Gemini on Vertex AI Express Mode using the Google Gen AI SDK.
• Intro to Grounding with Gemini in Vertex AI — Shows how to ground Gemini 3.5 Flash responses with Search, Enterprise Web Search, Maps, and Vertex AI Search.
• Intro to Logprobs — Introduces Gemini logprobs on Vertex AI for confidence, autocomplete, and RAG grounding analysis.
• Introduction to Long Context Window with Gemini on Vertex AI — Demonstrates Gemini long-context text, video, and audio prompts on Vertex AI with token counting.
• Getting Started with Model Optimizer — Introduces Vertex AI Model Optimizer routing with Google Gen AI SDK prompts and function calls.
• Intro to Agent Platform Multimodal Datasets — Builds Agent Platform multimodal datasets for Gemini tuning validation, resource estimates, tuning, and batch prediction.
• Question Answering with Generative Models on Vertex AI — Shows prompt patterns for Gemini question answering on Vertex AI, including simple fuzzy evaluation.
• Text Classification with Generative Models on Vertex AI — Classifies text with Gemini on Vertex AI using zero-shot, few-shot, and evaluation workflows.
• Text Extraction with Generative Models on Vertex AI — Uses Gemini on Vertex AI to extract structured facts from text with constrained and few-shot prompts.
• Get Started with Vertex AI Prompt Optimizer - Long prompt — Runs Vertex AI Prompt Optimizer to improve a long Gemini prompt using data-driven evaluation.
• Vertex AI RAG Engine with Vertex AI Feature Store — Builds a Vertex AI RAG Engine corpus backed by Vertex AI Feature Store and queries it with Gemini.
• Vertex AI RAG Engine with Vertex AI Vector Search — Builds a Vertex AI RAG Engine corpus backed by Vertex AI Vector Search and queries it with Gemini.
• Vertex AI RAG Engine with Vertex AI Search — Build a Vertex AI RAG Engine corpus backed by Vertex AI Search and query it with Gemini.
• Vertex AI RAG Engine with Weaviate — Creates a Vertex AI RAG Engine corpus backed by Weaviate and queries it with Gemini.
• Intro to thought signatures — Shows how Gemini thought signatures preserve reasoning context across multi-turn function calling.
• Intro to thought signatures with REST API — Shows how to pass Gemini thought signatures through REST function-calling turns.
• AI-Assisted Data Science Workflows in BigQuery — Builds a BigQuery multimodal housing workflow with Gemini enrichment, BQML clustering, and vector search.
• Analyzing movie posters in BigQuery with Gemini — Analyzes movie poster images in BigQuery with Gemini, embeddings, and vector search.
• Semantic Analysis in BigQuery with AI Functions — Uses BigQuery AI functions with Gemini to rank, classify, filter, join, and enrich pet product data.
• Introduction to Generative AI functions in BigQuery — Introduces BigQuery generative AI functions for SQL-based text analysis and forecasting.
• Test Document AI Gemini — Compares Document AI entity extraction with Gemini-based extraction on a PDF.
• Analyze Multimodal Data in BigQuery — Shows BigQuery multimodal analysis over structured tables and GCS media using ObjectRefs and Gemini.
• Analyze a codebase with Gemini in Vertex AI — Uses Gemini 3.5 Flash on Vertex AI to analyze, document, debug, and extend a GitHub codebase.
• Code Vulnerability Scanning & Automated Remediation using Gemini API in Vertex AI (Gemini 2.0) — Scans Python files from GCS with Gemini 2.0 Flash and exports vulnerability reports to CSV and JSON.
• Get hands-on with a customer support use case using Gemini and Gen AI SDK — Builds a Gemini customer-support flow for retail product matching, room-fit reasoning, tools, search, and Live API audio.
• Document Processing with Gemini — Processes PDFs with Gemini on Vertex AI for extraction, classification, QA, summarization, translation, and page selection.
• Patents Document Understanding with Gemini — Uses Gemini batch prediction on Vertex AI to extract structured fields and figure boxes from patent PDFs.
• Sheet Music Analysis with Gemini — Uses Gemini on Vertex AI to extract and edit sheet music PDF metadata and identify a song from audio.
• Text Summarization of Large Documents using LangChain 🦜🔗 — Summarizes large PDFs with LangChain and Gemini using stuff, map-reduce, and refine chains.
• Automating Income Taxes with Gemini — Classifies tax PDFs and extracts structured tax fields with Gemini on Vertex AI.
• Using Gemini in Education — Demonstrates Gemini education prompts across text, math, images, multiple images, and video.
• Gemini: An Overview of Multimodal Use Cases — Shows Gemini multimodal prompts for text, PDFs, images, video, audio, code, retail, diagrams, and comparisons.
• Building Knowledge Graphs with Gemini — Extracts knowledge graphs from text and PDF documents with Gemini using deterministic prompts and TSV outputs.
• Creative Content Generation with Gemini in Vertex AI and Imagen — Generates and personalizes GShoe marketing copy with Gemini, then outpaints product images with Imagen.
• Generating Consistent Imagery with Gemini 🍌 — Generates a consistent robot image story from one archive image using Gemini 2.5 Flash Image.
• Video Captioning with Gemini — Uses Gemini on Vertex AI to generate rich multimodal captions from a video in Cloud Storage.
• Video Data Curation - Video Quality Filtering — Filters video datasets by metadata, OCR/watermarks, aesthetics, and motion scores for curation.
• Multimodal Sentiment Analysis with Gemini — Compares Gemini sentiment analysis on audio versus a transcript of the same conversation.
• Slide Generation with Gemini and Marp — Uses Gemini and Marp to turn a blog post into a Markdown slide deck and PDF.
• Multimodal retail recommendation: using Gemini to recommend items based on images and image reasoning — Builds a Gemini multimodal retail recommender that explains chair fit for a room image.
• Product attributes extraction and detailed descriptions from images using Gemini 2.0 — Extract product attributes and detailed retail descriptions from images with Gemini 2.0 on Vertex AI.
• Detecting and Editing Visual Objects with Gemini — Detects visual objects with Gemini, crops them, and edits them with Gemini image models.
• Spatial understanding with Gemini 3 — Uses Gemini 2.5 Flash on Vertex AI to detect objects and draw normalized bounding boxes on images.
• Unlocking Multimodal Video Transcription with Gemini — Uses Gemini to build multimodal video transcripts with voice IDs, speaker names, and structured JSON output.
• Identifcation of Scene Transitions in Movies Using Gemini — Uses Gemini to detect movie scene transitions from a video and matching VTT subtitles.
• YouTube Video Analysis with Gemini — Analyzes public YouTube videos with Gemini for summaries, structured JSON extraction, and cross-video insights.
• Enhanced Vision Assistant with Gemini — Builds a Gemini vision assistant that detects objects, assesses hazards, and speaks navigation guidance.

SDK (1)

• Getting started with Google Generative AI using the Gen AI SDK — Introduces Google Gen AI SDK on Vertex AI for Gemini prompts, tools, caching, batches, and embeddings.

Vision (1)

• Veo 3.1 Video Generation — Generates Veo 3.1 videos from text and images with optional Gemini prompt optimization.

Related concepts

• Applied Use Cases · Function Calling & Tools · Getting Started · Vision · Prompt Engineering · RAG & Grounding · Audio & Speech · Embeddings & Vector Search

---

Part of the Generative AI Google Brain · Gemini Capabilities - Best Practices