Generative AI Google Brain

Gemini Capabilities - Best Practices

29 min read

Gemini Capabilities — Best Practices

Distilled from 81 notebooks tagged Gemini Capabilities in the GoogleCloudPlatform/generative-ai repository. The From the notebooks section below cites the per-notebook source for grounding.

Do this

  • Keep model configuration explicit per request with GenerateContentConfig, system_instruction, safety_settings, tools, response_schema, response_mime_type, generation parameters, and model-selection or thinking controls.
  • Use structured output for downstream systems by setting response_mime_type to application/json, defining response_schema or response_json_schema, marking required fields, allowing nullable fields when context may be missing, and validating parsed results.
  • Use low temperature or temperature=0 for deterministic extraction, classification, tool-calling, and code-execution examples; keep Gemini 3+ sampling defaults when the notebook recommends default temperature behavior.
  • Preserve full model response content in manual function-calling or tool histories so thought signatures are retained; prefer SDK chat history or automatic function calling when possible.
  • Declare tools with clear FunctionDeclaration names, descriptions, typed parameters, required fields, and allowed function modes; execute tools only in trusted application code and verify function call names before dispatch.
  • Use grounding when factuality matters: inspect grounding metadata, citations, search queries, retrieval queries, and grounding chunks, and choose Google Search, Enterprise Web Search, Maps, Vertex AI Search, or internal context based on compliance and data needs.
  • Count or compute tokens before long-context or large multimodal requests, tune media_resolution for cost and latency, and use context caching for repeated large prefixes or shared documents.
  • Use context caching deliberately: put large common content at the beginning of prompts, verify cached_content_token_count, set ttl or expire_time, remember caches are model-specific, and delete cached content when unused.
  • For multimodal inputs, use Part.from_uri or file_data with explicit MIME types for Cloud Storage/public URIs and Part.from_bytes or inline_data for local files; include text context so media is grounded in user intent.
  • For embeddings and retrieval, match query and document dimensionality, use the correct task type such as RETRIEVAL_QUERY, RETRIEVAL_DOCUMENT, or CLUSTERING, and rank with efficient dot product or cosine similarity.
  • For batch and pipeline workflows, use timestamped output locations, poll jobs until terminal states, sample large outputs from BigQuery, persist intermediate tables, and add notification or monitoring logic when built-in completion alerts are unavailable.
  • For memory-enabled agents, scope memories with a unique USER_ID, preload relevant memory at the start of a turn, explicitly add completed sessions to Memory Bank, and instruct the agent not to invent missing memories.
  • For speech generation, choose Cloud Text-to-Speech when output encoding or bidirectional streaming matters, choose Agent Platform when already using Gemini-TTS there, and assign voices with multi_speaker_voice_config or per-line synthesis as required.
  • Evaluate model behavior with reference data, custom task metrics, pairwise or pointwise Vertex AI Evaluation, schema-validity checks, extraction accuracy, DeepDiff-style inspection, and repeated model comparisons over the same inputs.
  • Clean up billable resources such as Agent Engines, batch jobs, context caches, Vertex AI experiments, Cloud Storage buckets, BigQuery datasets, pipelines, data stores, and prompt optimizer artifacts after tutorials or experiments.

Avoid this

  • Running notebooks before enabling the required APIs or configuring authentication, project ID, location, quota project, staging bucket, service accounts, or environment variables.
  • Assuming all features work globally or in every region; Text-to-Speech voices, Gemini-TTS, context caching, BigQuery datasets, Enterprise Web Search, batch prediction, tuning, and model access can be region- or allowlist-dependent.
  • Mixing incompatible thinking controls, such as using Gemini 3 thinking_level with legacy thinking_budget, or assuming thinking can be turned off for models where it cannot.
  • Dropping thought signatures or previous candidate content during manual function calling, which can cause later API requests to fail with missing thought_signature errors.
  • Treating model tool calls as automatically executed; the application must run external functions, handle parallel or chained calls, and send function responses back to the model.
  • Ignoring finish_reason, safety_ratings, missing candidates, or response.text being None when safety filters block content.
  • Using mismatched schemas, optional fields, unsupported OpenAPI schema fields, missing required fields, or exact string matching for structured output evaluation when custom accuracy metrics are more robust.
  • Exceeding modality limits or using unsupported input formats, such as too many images, long PDFs, long audio/video, inaccessible Cloud Storage/public URIs, missing MIME types, or incompatible embedding dimensionality.

From the notebooks

Building a Multimodal Trip Planner with ADK on Vertex AI Agent Engine Memory Bank

  • Use managed topics for common memory categories and custom topics for domain-specific memory extraction.
  • Provide text context alongside file_data so multimodal memories are grounded in user intent.
  • Scope generated memories with user_id.
  • Use PreloadMemoryTool so relevant memories are fetched at the start of a conversation turn.
  • Instruct the agent not to make up facts when memories are unavailable.

Intro to Gemini Data Analytics

  • Provide critical context for the agent through system_instruction.
  • Attach example_queries and glossary_terms when using BigQuery datasources.
  • Attach looker_golden_queries only when Looker is selected and the flag is enabled.
  • Use updateMask when patching data agent fields.
  • Route streamed system messages by type: text, schema, data, chart, and error.

Get started with Gemini-TTS voices using Text-to-Speech

  • Choose Cloud Text-to-Speech API when specific output encodings or bidirectional streaming are needed.
  • Choose Agent Platform API when already using Gemini-TTS from AI Studio or other Agent Platform models.
  • Use natural-language prompts to steer style, accent, pace, tone, and emotional expression.
  • Use multi_speaker_voice_config to assign speaker aliases to specific voices.
  • Stream audio chunks immediately to the frontend in real-time applications.

Create a Multi-Speaker Podcast with Gemini 2.0 & Text-to-Speech

  • Use controlled generation with response_mime_type set to application/json and a response_schema.
  • Parse Gemini output with json.loads and handle JSONDecodeError and KeyError.
  • Skip audio synthesis when no dialogue is generated.
  • Use a regional Text-to-Speech API endpoint based on TTS_LOCATION.
  • Keep speaker labels constrained to R and S for MultiSpeakerMarkup turns.

Narrate a Multi-character Story with Gemini and Text-to-Speech

  • Use response_schema with Gemini to produce structured output for downstream processing.
  • Use a narrator voice for scene settings and title narration.
  • Filter available voices before assigning them to characters.
  • Use one Text-to-Speech call per dialogue line when different voices are required.
  • Combine generated clips with short silence between lines and remove intermediate MP3 files.

Introduction to Gemini Multimodal Embeddings

  • Embed multiple text prompts in one API call for efficiency.
  • Use output_dimensionality to reduce storage cost and improve search speed when lower dimensions are acceptable.
  • Choose input structure deliberately: one Content object aggregates parts into one embedding, while multiple contents return separate embeddings.
  • Create post-level multimodal representations by aggregating separate embeddings, for example by averaging.
  • Truncate PDFs before embedding when they exceed the six-page limit.

Anomaly Detection of Infrastructure Logs using Gemini and BigQuery Vector Search

  • Group raw logs by a meaningful session identifier before summarization.
  • Use a one-shot prompt and instruct Gemini not to repeat raw logs or speculate.
  • Persist intermediate outputs in BigQuery tables so long-running jobs can resume after failures.
  • Batch ML.GENERATE_TEXT calls to reduce loss from timeouts or quota exhaustion.
  • Use CLUSTERING task_type for embeddings when the downstream task is clustering-based anomaly detection.

Get started with Vertex AI Memory Bank - ADK

  • Use a unique USER_ID to scope memories to a particular user.
  • Use PreloadMemoryTool so the agent can retrieve relevant user context.
  • Prompt the agent to personalize responses and naturally reference past conversations when relevant.
  • Retrieve the completed session before adding it to Memory Bank.
  • Delete the Agent Engine after experimentation to avoid charges.

Semantic Router Agent

  • Use a structured Pydantic schema for router output instead of parsing free-form text.
  • Keep router temperature low and seed fixed for more deterministic classification.
  • Store user and model messages per turn so later calls can include conversation history.
  • Route unsupported inputs to a fallback response instead of an expert node.
  • Use LangGraph MemorySaver with thread_id to preserve session-specific state.

Intro to Batch Inference with the Gemini API

  • Use batch inference for large input sets that are not latency sensitive.
  • Specify explicit Cloud Storage or BigQuery output locations for completed predictions.
  • Check batch job status with client.batches.get before retrieving results.
  • List jobs with client.batches.list when inspecting project batch activity.
  • Delete created batch prediction jobs during cleanup.

Monitor batch prediction with Gemini API

  • Initialize Vertex AI with project, location, and staging_bucket before running the pipeline.
  • Use timestamped BigQuery output table names to avoid collisions across batch runs.
  • Wrap the batch job in a Vertex AI Pipeline and use VertexNotificationEmailOp because Gemini batch prediction lacks built-in completion notifications.
  • Poll BatchPredictionJob until has_ended and fail the component when has_succeeded is false.
  • Sample prediction results from BigQuery instead of loading the whole output table.

Using OpenAI libraries with Gemini on Vertex AI

  • Use google-auth cloud-platform scoped credentials instead of static API keys.
  • Use a regional Vertex AI endpoint when using context caching.
  • Use tool_choice to control whether the model may call tools.
  • Use response schemas with Pydantic when structured output is required.
  • Use context caching for repeated large inputs to reduce tokens sent and lower request cost.

Intro to Gemini Agentic Vision

  • Enable code execution when the model needs to crop, inspect, calculate, plot, or draw instead of guessing from a static image.
  • Parse response parts explicitly to inspect reasoning text, generated code, execution output, and resulting images.
  • Use PIL and IPython display to render images returned from code execution.
  • Use structured image parts from bytes or URI with the correct MIME type.

Intro to Generating and Executing Python Code with Gemini 3

  • Use the Google Gen AI SDK unified interface with Vertex AI for enterprise-ready projects.
  • Pass code_execution as a Tool instead of expecting code execution by default.
  • Inspect executable_code and code_execution_result response parts separately.
  • Save or display the output code, result, or outcome downstream in the application.
  • Use chat sessions when code needs to be rewritten iteratively with history.

Intro to Computer Use with Gemini

  • Run Computer Use agents in a secure controlled environment such as a sandboxed VM, container, or dedicated browser profile.
  • Implement client-side action handling and screenshot capture.
  • Append both model responses and function responses to conversation history.
  • Return a FunctionResponse for each executed action, including parallel actions.
  • Handle missing candidates because safety filters may return no candidates.

Intro to Context Caching with the Gemini API

  • Put large and common contents at the beginning of the prompt to increase implicit cache hit chances.
  • Send requests with similar prefixes in a short amount of time for implicit caching.
  • Use usage_metadata.cached_content_token_count to verify cached token usage.
  • Use cached_content.name or resource_name to reference explicit cached content.
  • Set ttl or expire_time to control cache expiration.

Intro to Structured Output with the Gemini API

  • Set response_mime_type to application/json for JSON outputs.
  • Use response_schema with Pydantic or OpenAPI-style schemas for controlled JSON generation.
  • Use response_json_schema when JSON Schema conditional logic is needed.
  • Use response.parsed to consume structured outputs as objects or dictionaries.
  • Use text/x.enum when the model must choose from predefined enum values.

Enhancing quality and explainability with Vertex AI Evaluation

  • Generate multiple candidate responses before ranking for quality.
  • Use pairwise evaluation to select the best response among candidates.
  • Use pointwise evaluation to report quality and groundedness explanations for the selected response.
  • Use a prompt template that includes both instruction and context during evaluation.
  • Use unique experiment_run_name values with uuid.uuid4().

Evaluate Gemini Structured Output

  • Use structured output with response_mime_type application/json and response_schema for consistent JSON.
  • Keep reference ground truth alongside each model response in the evaluation dataset.
  • Evaluate both schema validity and extraction accuracy with custom metrics.
  • Use DeepDiff to inspect field-wise differences between reference and response.
  • Compare multiple Gemini model ids over the same prompt and input images.

Forced Function Calling with Tool Configurations in Gemini

  • Use typed FunctionDeclaration parameters with Schema and Type values.
  • Set temperature=0 for deterministic function-calling examples.
  • Use AUTO when the model should decide whether a tool is needed.
  • Use ANY with allowed_function_names to force a specific function or subset of functions.
  • Check response.function_calls[0].name before executing the external function.

Multimodal Function Calling with the Gemini API & Python SDK

  • Use FunctionDeclaration parameters to constrain predicted function arguments to a JSON schema.
  • Group function declarations into Tool objects before passing them to Gemini.
  • Set temperature=0 for deterministic function-call prediction examples.
  • Use Part.from_uri with the correct MIME type for image, video, audio, and PDF inputs.
  • Save the model response content when returning a later tool response with the thought signature.

Working with Parallel Function Calls and Multiple Function Responses in Gemini

  • Use FunctionDeclaration parameters to describe callable functions clearly.
  • Wrap function declarations in a Tool and pass it through GenerateContentConfig.
  • Use temperature=0 for these tool-calling examples.
  • Extract structured function calls from response.function_calls before executing application code.
  • Fan out independent external API calls in application code, then return all function responses to Gemini in bulk.

Intro to Gemini 2.5 Flash

  • Use a Google Cloud project with Vertex AI API enabled for most users.
  • Use streaming generation when responses should appear as they are generated.
  • Set thinking_budget based on task complexity to manage quality and speed.
  • Set thinking_budget to 0 for simpler examples where lower latency is desired.
  • Inspect usage_metadata, thoughts_token_count, and total_token_count when evaluating thinking cost.

Intro to Gemini 2.5 Flash-Lite

  • Use GenerateContentConfig to keep model parameters, system instruction, safety settings, tools, and schemas explicit per request.
  • Use system_instruction to provide task context, persona, formatting rules, and interaction guidelines.
  • Use safety_settings to adjust blocking behavior for specific harm categories.
  • Inspect usage_metadata to see prompt, candidate, thoughts, and total token counts.
  • Use response_schema with response_mime_type=“application/json” for controlled generation.

Gemini 2.5 Flash Image (Nano Banana 🍌) Generation

  • Set response_modalities to match the desired output modalities.
  • Use ImageConfig to specify aspect_ratio for generated images.
  • Check finish_reason before assuming an image was generated.
  • Iterate over response content parts and handle both text and inline_data.
  • Use Part.from_bytes for local image inputs.

Intro to Gemini 2.5 Pro

  • Use a Google Cloud Project for authentication for most users.
  • Set thinking_budget to control quality and speed of response.
  • Inspect usage_metadata token counts when evaluating thinking budget behavior.
  • Use include_thoughts only when summarized thoughts are needed.
  • Use lower thinking budget for examples that do not need extra reasoning to reduce latency.

Intro to Gemini 3.1 Flash-Lite

  • Use thinking_level MINIMAL for low-complexity, lower-latency responses.
  • Use higher thinking levels for tasks requiring deeper reasoning.
  • Use Google Gen AI SDK chat history or append the full model response so thought signatures are handled automatically.
  • Append response.candidates[0].content when doing manual function calling across turns.
  • Set media_resolution based on detail needs and input length.

Intro to Gemini 3.1 Pro

  • Use Google Gen AI SDK chat history or append the full model response so thought signatures are handled automatically.
  • Set thinking_level to low for simple instruction following or chat when lower latency is desired.
  • Use high thinking level for non-trivial reasoning tasks such as concurrency-safe code generation.
  • Set media_resolution per media part when different images or videos need different detail levels.
  • Lower media resolution for very long inputs to manage token count.

Intro to Gemini 3.5 Flash

  • Use streaming for user-facing or chat-based interfaces to receive chunks immediately.
  • Use async generation to improve throughput in web apps and multi-agent systems.
  • Use per-part media resolution when different media inputs need different fidelity levels.
  • Use automatic function calling when possible because the SDK manages thought signatures behind the scenes.
  • Preserve response.candidates[0].content in manual tool histories to maintain reasoning context.

Intro to Gemini 3 Flash (Preview)

  • Use the Google Gen AI SDK chat history or append the full model response so thought signatures are handled automatically.
  • Set thinking_level to MINIMAL or LOW for lower-complexity tasks where lower latency is preferred.
  • Use HIGH thinking_level for non-trivial reasoning tasks such as the double-checked locking C++ example.
  • Lower media_resolution for long videos or extensive documents to fit token limits.
  • Keep Gemini 3 temperature at its default value of 1.0 as recommended in the notebook.

Getting Started with Chat with Gemini

  • Use environment variables for project and region fallback.
  • Set system instructions when creating GenAI SDK chat sessions.
  • Inspect response metadata including safety_ratings and usage_metadata.
  • Retrieve chat history when validating stateful behavior.
  • Add prior chat history as alternating user and model messages.

REST API

  • Set GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION before building the API endpoint.
  • Use the global aiplatform.googleapis.com host unless a regional location is configured.
  • Set generation parameters such as temperature, top_p, top_k, max_output_tokens, candidate_count, and stop_sequences deliberately.
  • Use safety_settings to configure harm blocking thresholds.
  • Specify role values for multi-turn chat content.

Getting started with Gemini using Vertex AI in Express Mode

  • Use streaming responses when reducing perceived latency matters.
  • Use system instructions to steer model behavior for a task.
  • Inspect finish_reason and safety_ratings when using safety settings.
  • Use response_schema and response_mime_type for structured JSON output.
  • Use count_tokens before sending a request to estimate input token usage.

Intro to Grounding with Gemini in Vertex AI

  • Compare ungrounded and grounded responses to show the effect of grounding.
  • Use grounding metadata to display citations, grounding chunks, search queries, and retrieval queries.
  • Use Enterprise Web Search for web grounding when compliance requirements make Google Search logging unsuitable.
  • Use Vertex AI Search for internal documents that are not available on the public internet.
  • Delete projects, data stores, and disable APIs after the tutorial to avoid charges.

Intro to Logprobs

  • Use response_logprobs=True only when token confidence data is needed.
  • Use logprobs to inspect top alternative tokens and debug model behavior.
  • Constrain classification outputs with response_schema and text/x.enum before comparing label confidence.
  • Flag classifications for human review when top choices have close log probabilities.
  • Convert log probabilities with math.exp before applying probability thresholds.

Introduction to Long Context Window with Gemini on Vertex AI

  • Start by putting all relevant tokens into the context window when the input fits, while recognizing RAG and summarization can still be relevant.
  • Use count_tokens() before generate_content() for long-context requests.
  • Use Part.from_uri with explicit MIME types for text, video, and audio files.
  • Inspect response.usage_metadata after generation.
  • Use context caching to reduce time and cost for repeated large-context requests.

Getting Started with Model Optimizer

  • Use environment variables for project and region fallbacks.
  • Set routing preferences explicitly with ModelSelectionConfig.
  • Keep system instruction and model selection config inside GenerateContentConfig.
  • Use streaming when incremental response display is desired.
  • Declare function parameters and response schemas before passing tools to the model.

Intro to Agent Platform Multimodal Datasets

  • Use BigFrames instead of pandas for larger datasets.
  • Inspect created datasets with resource name, display name, BigQuery URI, and BigFrames preview.
  • Attach the read configuration to the dataset before tuning and batch prediction workflows.
  • Run tuning validity assessment before starting supervised fine-tuning.
  • Estimate tuning resources before running a tuning job.

Question Answering with Generative Models on Vertex AI

  • Use specific, context-rich, grammatically correct prompts for question answering.
  • Experiment with generation parameters such as temperature and max_output_tokens.
  • Use few-shot examples when the desired answer style is specific or short.
  • For closed-domain Q&A, provide the knowledge base as prompt context.
  • Tell the model to return “Information not available in provided context” when the context lacks the answer.

Text Classification with Generative Models on Vertex AI

  • Use explicit candidate labels or task instructions in classification prompts.
  • Use few-shot examples when the model should follow a specific class mapping or answer format.
  • Use low temperature and limited max output tokens for concise classification responses.
  • Evaluate classification outputs against ground truth labels when they are available.
  • Keep custom model training separate when prompt-based classification is sufficient for the notebook scope.

Text Extraction with Generative Models on Vertex AI

  • Initialize Vertex AI with an explicit project and location.
  • Use low temperature for extraction-oriented prompts.
  • Ask for JSON format when downstream systems need structured output.
  • Constrain answers to provided text for troubleshooting responses.
  • Use few-shot prompting to guide output organization.

Get Started with Vertex AI Prompt Optimizer - Long prompt

  • Use placeholders to freeze static prompt sections while optimizing the rest of a long system instruction.
  • Provide examples where the current system instruction performs poorly.
  • Use 50-100 distinct samples for reliable prompt optimization results.
  • Use a target column for computation-based metrics such as question_answering_correctness.
  • Use response_schema and response_mime_type to constrain integer JSON output.

Vertex AI RAG Engine with Vertex AI Feature Store

  • Use the required BigQuery schema fields for the RAG Feature Store source table.
  • Use optimized online serving for FeatureOnlineStore when using vector similarity search.
  • Use chunk_size and chunk_overlap when importing files into the RAG corpus.
  • List imported files after import because processing may take a few seconds.
  • Run FeatureView sync after uploading data to make it available for online serving.

Vertex AI RAG Engine with Vertex AI Vector Search

  • Use environment variables for project and region defaults when notebook parameters are not provided.
  • Create a public Vector Search index endpoint because RAG Engine supports public endpoints.
  • Tune chunk_size and chunk_overlap during file import.
  • Use similarity_top_k and vector_distance_threshold to control retrieved context.
  • List files after import to check ingestion progress.

Vertex AI RAG Engine with Vertex AI Search

  • Use environment variables for GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION when notebook parameters are not provided.
  • Use Vertex AI Search as the retrieval backend for large datasets, low-latency retrieval, and improved scalability.
  • Import documents from Cloud Storage using INCREMENTAL reconciliation mode.
  • Check the created corpus with rag.get_corpus before querying.
  • Clean up created RAG resources with rag.delete_corpus when delete_rag_corpus is enabled.

Vertex AI RAG Engine with Weaviate

  • Use GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION environment variables as fallbacks for project and region.
  • Configure a Google first-party embedding model for the corpus.
  • Set chunk_size and chunk_overlap when importing files.
  • Check created resources with rag.get_corpus and rag.list_files.
  • Use similarity_top_k and vector_distance_threshold to control retrieved context.

Intro to thought signatures

  • Pass the model’s previous response content back into contents so the thought signature is preserved.
  • Send function execution results as role=“tool” with Part.from_function_response.
  • Group related function declarations in a Tool so the model can choose the needed function.
  • Use thought signatures for multi-turn interactions with external tools that require reasoning context.

Intro to thought signatures with REST API

  • Set GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION before constructing the endpoint.
  • Use function_declarations with explicit parameters and required fields.
  • Include thought signatures when sending function execution results back to the model.
  • Keep the full conversation history when requesting the final response.
  • Clean up generated response*.json files after the tutorial.

AI-Assisted Data Science Workflows in BigQuery

  • Create a dedicated BigQuery dataset to contain tutorial tables, views, and models.
  • Use ML.DESCRIBE_DATA for quick exploratory statistics before modeling.
  • Filter raw data and add engineered features before AI enrichment and clustering.
  • Define an explicit AI.GENERATE_TABLE output schema for reliable typed columns.
  • Inspect enriched rows and generated embeddings before downstream modeling or indexing.

Analyzing movie posters in BigQuery with Gemini

  • Create a dedicated dataset for the demo resources.
  • Use a Cloud resource connection for BigQuery access to Cloud Storage objects.
  • Use output_schema with AI.GENERATE when extracting structured fields.
  • Store intermediate Gemini analysis and embedding results in BigQuery tables.
  • Use public datasets and joins to enrich generated analysis with structured metadata.

Semantic Analysis in BigQuery with AI Functions

  • Use managed AI functions for analysts who want prompt optimization handled by BigQuery.
  • Use AI.SCORE for semantic ranking by subjective criteria.
  • Use AI.CLASSIFY with explicit categories and a fallback category such as All Pets.
  • Use AI.IF for semantic filtering in WHERE clauses and semantic joins in JOIN ON clauses.
  • Use AI.GENERATE_BOOL for SELECT-clause enrichment when full prompt control is needed.

Introduction to Generative AI functions in BigQuery

  • Create a Cloud resource connection for BigQuery access to Vertex AI services.
  • Grant the connection service account the required BigQuery connection and Vertex AI roles.
  • Wait for IAM changes to propagate before running model calls.
  • Use model_params to control temperature, maxOutputTokens, and thinking_budget.
  • Use output_schema when generated results need typed structure.

Test Document AI Gemini

  • Process the same input document through both extraction paths before comparing outputs.
  • Use separate prompts for extraction and comparison.
  • Upload the document to temporary Cloud Storage for model-based extraction, then delete the temporary file.
  • Keep Document AI configuration values explicit: project, location, processor, file path, and MIME type.

Analyze Multimodal Data in BigQuery

  • Use ObjectRefs as secure pointers instead of storing unstructured file bytes in BigQuery tables.
  • Use object tables when every file in a GCS path should automatically become an ObjectRef row.
  • Use OBJ.MAKE_REF and OBJ.FETCH_METADATA when URI columns already exist in structured tables.
  • Join structured records with ObjectRef columns to create a reusable multimodal table.
  • Define output_schema for AI.GENERATE and AI.GENERATE_TABLE to return typed, queryable results.

Analyze a codebase with Gemini in Vertex AI

  • Use a shallow clone when loading the GitHub repository.
  • Create a context cache for the codebase to avoid resending the full context on every request.
  • Include a system instruction that frames the model as a coding expert for code-related questions.
  • Use streaming generation for longer developer guides, troubleshooting guides, and recommendations.
  • Separate tool-based GitHub issue retrieval from cached-content prompts because tools cannot be added at runtime with cached content.

Code Vulnerability Scanning & Automated Remediation using Gemini API in Vertex AI (Gemini 2.0)

  • Add each filename as a separator before its code so the model can identify findings per file.
  • Use a clear prompt that specifies vulnerability name, description, recommendations, and recommended code.
  • Set generation parameters including temperature, top_p, top_k, candidate_count, and max_output_tokens.
  • Export findings to CSV and JSON for further analysis, benchmarking, and integration with security tools.
  • Treat the workflow as experimental and continue validation before using it as a robust security tool.

Get hands-on with a customer support use case using Gemini and Gen AI SDK

  • Use system_instruction to give task role and response guidance across the interaction.
  • Use Part.from_uri with explicit MIME types for image inputs.
  • Use response_schema and response_mime_type=“application/json” for downstream structured processing.
  • Use temperature=0 when calling Google Search for store-location lookup.
  • Return function results with Part.from_function_response so the model can incorporate external data.

Document Processing with Gemini

  • Use explicit system instructions for extraction, classification, QA, and summarization tasks.
  • Use response_schema and response_mime_type to enforce structured JSON or enum outputs.
  • Use Pydantic models to define extraction schemas with field descriptions.
  • Use Part.from_uri for GCS or HTTPS PDFs and Part.from_bytes for local PDF bytes.
  • Use response.parsed when consuming structured model outputs.

Patents Document Understanding with Gemini

  • Use a detailed prompt plus responseMimeType application/json and responseSchema for controlled generation.
  • Include the PDF as fileData with mimeType application/pdf instead of extracting document text separately.
  • Use a systemInstruction to set the model role as an expert at analyzing patent documents.
  • Write batch requests to BigQuery and let the batch job return outputs to a BigQuery table.
  • Poll batch job state before reading destination results.

Sheet Music Analysis with Gemini

  • Use environment variables for GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION when notebook parameters are not provided.
  • Set safety settings and a domain-specific system instruction before generation.
  • Use Part.from_uri with explicit MIME types for PDF and audio inputs.
  • Use response_mime_type=“application/json” before parsing model output with json.loads.
  • Copy PDF pages into a writer before adding metadata and saving the file.

Text Summarization of Large Documents using LangChain 🦜🔗

  • Use environment variables as fallbacks for Google Cloud project and region configuration.
  • Keep summarization instructions in PromptTemplate objects for stuffing, map, combine, question, and refine prompts.
  • Test stuffing on a small page subset and handle exceptions from oversized context.
  • Use map_reduce or refine when document length exceeds what stuffing can handle.
  • Set return_intermediate_steps=True to inspect chunk-level outputs.

Automating Income Taxes with Gemini

  • Use temperature=0 for deterministic classification and extraction.
  • Constrain classification with an Enum response schema.
  • Use document-specific Pydantic schemas for typed structured extraction.
  • Map classified document types to their extraction schemas before calling the model.
  • Normalize parsed JSON before joining it back into the source DataFrame.

Using Gemini in Education

  • Use a low default temperature for more consistent responses.
  • Use few-shot examples to guide response structure and formatting for text correction.
  • Ask for step-by-step reasoning to reduce hallucinations in math tasks.
  • Use structured output requests such as JSON lists or tables when asking detailed questions.
  • Ask video questions using the video only and request timestamps plus source type such as image, text, or speech.

Gemini: An Overview of Multimodal Use Cases

  • Pass non-text inputs with explicit MIME types using Part.from_uri.
  • Use context caching for repeated questions over a large codebase instead of resending the same prompt.
  • Set temperature to 0 for factual identification prompts, as shown for the train-line example.
  • Label candidate images in the prompt when asking the model to choose among provided images.
  • Tell the model not to make up information when answers must be grounded only in attached audio or video.

Building Knowledge Graphs with Gemini

  • Use only the provided input data to avoid relying on memorized general knowledge.
  • Use explicit domain terminology such as entities, relationships, nodes, and edges.
  • Include deterministic settings such as temperature=0.0, top_p=0.0, and seed=42 for extraction tasks.
  • Separate data schema, extraction instructions, and output format in the prompt.
  • Use TSV for table-shaped outputs when token efficiency matters.

Creative Content Generation with Gemini in Vertex AI and Imagen

  • Add product description to improve generated output quality.
  • Use product images as multimodal context for marketing messages.
  • Target generated posts to specific platforms such as Facebook, Instagram, LinkedIn, and Twitter.
  • Reuse generated posts as reference context when personalizing for audience segments.
  • Prepare an expanded base image and mask before calling Imagen outpainting.

Generating Consistent Imagery with Gemini 🍌

  • Use environment variables or Colab Secrets instead of hardcoding API configuration.
  • Use a character sheet as a reusable design reference for future image-generation tasks.
  • Refer explicitly to input images, such as Image 1 and Image 2, to avoid ambiguity.
  • Clarify removed or changed objects, such as no longer holding the map or removing ice axes.
  • Spend time refining the first scene because it cascades into later generated scenes.

Video Captioning with Gemini

  • Use a detailed system prompt that specifies perspective, camera movement, subject motion, setting, lighting, body language, and visible text.
  • Return only the caption by instructing the model not to send a preamble.
  • Count video tokens separately when token accounting matters.
  • Return usage metadata, model version, generated text, and video-only token count for traceability.
  • Review the sample video before captioning to understand visual, textual, and audio elements.

Video Data Curation - Video Quality Filtering

  • Discard low-quality clips to use limited modeling compute efficiently.
  • Use metadata filters such as FPS, duration, resolution, brightness, and aspect ratio for downstream modeling requirements.
  • Use structured JSON output for separating detected text from watermark descriptions.
  • Set Gemini temperature to 0.0 for deterministic extraction-style calls.
  • Manually inspect low-scoring videos to choose an aesthetic score threshold.

Multimodal Sentiment Analysis with Gemini

  • Set project and location before creating the Gen AI client.
  • Use Part.from_uri with an explicit MIME type for Cloud Storage audio inputs.
  • Compare direct audio analysis with transcript-based analysis to understand modality differences.
  • Use speaker labels in prompts for conversation analysis.
  • Set response_modalities to TEXT when requesting text-only output.

Slide Generation with Gemini and Marp

  • Use PROJECT_ID from GOOGLE_CLOUD_PROJECT when the placeholder is not replaced.
  • Set a system instruction to restrict Gemini output to Markdown.
  • Provide the Marp repository as context so generated slides can follow Marp style and formatting.
  • Use max_output_tokens=8192 to allow enough room for a complete slide deck.
  • Write generated Markdown to slide-deck.md before rendering it with Marp CLI.

Multimodal retail recommendation: using Gemini to recommend items based on images and image reasoning

  • Combine text and image in a single prompt for visual understanding.
  • Label each candidate image within the prompt so the model can reference options clearly.
  • Provide the available item images when recommendations should be limited to store inventory.
  • Use response_mime_type=“application/json” when recommendations need to plug into an application.

Product attributes extraction and detailed descriptions from images using Gemini 2.0

  • Use temperature 0 for deterministic attribute extraction.
  • Provide system instructions that constrain answers to visible product evidence.
  • Use a closed vocabulary when attribute values must come from an approved set.
  • Parse model JSON output before returning it to application code.
  • Use debug mode to display the image and prompt during development.

Detecting and Editing Visual Objects with Gemini

  • Store environment configuration outside source code with environment variables, Colab Secrets, or platform defaults.
  • Use structured outputs with response_mime_type and response_schema for automated parsing.
  • Keep bounding box granularity controlled through prompt wording.
  • Use precise prompts such as excluding captions from boxes or preserving original line breaks.
  • Use low randomness, seed, and image response modalities for more deterministic restoration outputs.

Spatial understanding with Gemini 3

  • Use response_mime_type=“application/json” and response_schema=list[BoundingBox] for controlled generation.
  • Put repeated output-format rules in system_instruction to keep prompts shorter.
  • Use safety_settings with HARM_CATEGORY_DANGEROUS_CONTENT set to BLOCK_ONLY_HIGH.
  • Give repeated objects unique labels based on distinct characteristics such as color, size, or position.
  • Scale normalized box coordinates by image width and height before drawing overlays.

Unlocking Multimodal Video Transcription with Gemini

  • Use environment variables or Colab Secrets instead of hardcoding API configuration.
  • Start with simple prompts to observe Gemini’s natural behavior before refining instructions.
  • Craft prompts iteratively, precisely, and concisely.
  • Decouple transcripts and speakers into linked tables with a consistent voice ID.
  • Start transcript generation with audio-focused work before extracting speaker names from visual and audio cues.

Identifcation of Scene Transitions in Movies Using Gemini

  • Use temperature 0 for consistent scene boundary extraction.
  • Provide both video and VTT inputs to combine visual and dialogue cues.
  • Use response_mime_type application/json with response_schema for structured outputs.
  • Define explicit criteria for narrative, visual, dialogue, audio, and cohesion signals.
  • Do not count jump-cuts or insert shots as transitions unless they indicate meaningful narrative shifts.

YouTube Video Analysis with Gemini

  • Use Part.from_uri with mime_type=“video/webm” for public YouTube video inputs.
  • Use response_mime_type=“application/json” and response_schema for structured extraction.
  • Constrain enum fields in the response schema when only specific values are valid.
  • Prompt the model to use only information in the video itself for extraction.
  • Use asynchronous generation for analyzing multiple videos more efficiently.

Enhanced Vision Assistant with Gemini

  • Use environment variables for GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION when possible.
  • Allow an explicit credentials path but also support application default credentials.
  • Filter small detections with MIN_OBJECT_SIZE before generating guidance.
  • Prioritize navigation guidance by urgency using NavigationPriority.
  • Clean up camera/audio resources and temporary audio files in stop().

Getting started with Google Generative AI using the Gen AI SDK

  • Read project and region from GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION when notebook parameters are unset.
  • Use GenerateContentConfig for system instructions, parameters, safety settings, tools, schemas, and cached content.
  • Use Part.from_uri for model-readable Cloud Storage or HTTPS file inputs.
  • Inspect response.candidates[0].safety_ratings after applying safety filters.
  • Use response_mime_type application/json with response_schema for controlled generation.

Veo 3.1 Video Generation

  • Use detailed prompts for better video quality.
  • Include subject, action, scene, camera, style, temporal, audio, and dialogue details when relevant.
  • Use Gemini to synthesize keyword choices into a cohesive Veo prompt.
  • Use Veo 3.1 Fast when latency is a priority.
  • Use Veo 3.1 Lite for rapid prototyping and iteration.

Back to Gemini Capabilities · Best Practices Map

Graph View

Backlinks