Generative AI Google Brain

Applied Use Cases - Best Practices

31 min read

Applied Use Cases — Best Practices

Distilled from 88 notebooks tagged Applied Use Cases in the GoogleCloudPlatform/generative-ai repository. The From the notebooks section below cites the per-notebook source for grounding.

Do this

  • Start every notebook workflow by validating project, billing, API enablement, authentication, quota project, location, and required IAM roles before making model or service calls.
  • Use explicit system instructions, domain context, examples, glossary terms, golden queries, topic descriptions, and tool instructions so the model has the business and data context needed for reliable behavior.
  • Constrain model outputs with response schemas, JSON MIME types, OpenAPI-style function parameters, typed BigQuery AI functions, or output_schema when downstream code must parse or query the result.
  • Test tool functions, retrieval calls, BigQuery queries, and local agent behavior directly before wiring them into deployed agents, graphs, or multi-step orchestration.
  • Use low temperature or fixed seeds where the workflow needs repeatable routing, classification, tool arguments, search-agent behavior, or evaluation outputs, while still accounting for possible variation.
  • Ground answers in retrieved or connected data rather than pretrained knowledge for domain-specific workflows, and keep metadata such as page, chunk, source document, filename, or table context with retrieved content.
  • Use the same embedding model and appropriate task_type for related document and query embeddings; store dense and sparse representations when hybrid search is required.
  • Batch, poll, and retry long-running or quota-sensitive work such as Agent Engine deployments, Data Agent operations, tuning jobs, vector index creation, embedding generation, Gemini summarization, and A2A tasks.
  • Preserve existing IAM policies before setting updates, grant only required roles to connection or service accounts, and allow time for IAM propagation before dependent BigQuery or Vertex AI calls.
  • Persist intermediate results in BigQuery tables, Cloud Storage, or cached contexts so expensive or long-running workflows can be inspected, resumed, and reused.
  • Evaluate with multiple metrics and inspect row-level explanations or samples; compare models, prompt versions, retrieval behavior, tuned models, and custom autoraters on the target use case before recommending changes.
  • Use train, validation, and test splits for tuning workflows, validate chosen/rejected examples or JSONL records before launching jobs, and compare base and tuned responses on the same prompts.
  • Handle multiple response shapes defensively, including missing candidates, invalid JSON, safety-filtered responses, direct messages versus tasks, streamed text/schema/data/chart/error events, and missing audio or image data.
  • Mask secrets and tokens in logs, load API keys from environment variables or prompts, and avoid printing full credentials or voice keys in user interfaces.
  • Run cleanup sections for Agent Engines, endpoints, indexes, index endpoints, BigQuery datasets, connections, Cloud Storage buckets, experiments, and tuned deployments to avoid ongoing costs.

Avoid this

  • Forgetting to enable required APIs, billing, or authentication before running calls to Vertex AI, Agent Platform, BigQuery, Dataform, Text-to-Speech, Document AI, Vertex AI Search, or Gemini Data Analytics.
  • Mixing incompatible regions or assuming defaults: many notebooks use us-central1, global, US, or us connections, while speech, tuning, Agent Engine, BigQuery, and Looker workflows have specific regional constraints.
  • Missing IAM on service accounts or connections, especially roles/aiplatform.user, Storage Object Viewer or User, BigQuery connection roles, Discovery Engine permissions, or Agent Engine service-agent permissions.
  • Overwriting an existing data agent IAM policy by calling Set IAM Policy without first fetching and preserving the current policy.
  • Assuming long-running operations are immediate; deployments, tuning jobs, Data Agent creation, vector search deployment, indexing, embeddings, and large summarization jobs require polling and can take minutes to hours.
  • Assuming model output is always parseable or complete; JSON can be invalid, candidates can be absent, finish reasons can vary, audioContent can be missing, and A2A message sends can return either tasks or direct messages.
  • Leaving billable resources active after tutorials, including deployed Agent Engines, tuned endpoints, Vector Search resources, BigQuery datasets, connections, experiments, Cloud Storage buckets, and vector indexes.
  • Relying on semantic search alone for proprietary terms, new product names, SKUs, or out-of-domain identifiers when hybrid dense plus sparse retrieval is more appropriate.

From the notebooks

Customizing Memory Topics

  • Create a baseline with default managed topics before adding custom topics.
  • Use the same conversation for default and custom engines to make the comparison apples-to-apples.
  • Keep custom topics focused on separate domain dimensions to reduce overlap and improve precision.
  • Write detailed topic descriptions with examples and exclusions for what belongs in other topics.
  • Use realistic few-shot conversations that show the desired memory extraction style.

Governance with Vertex AI Memory Bank

  • Use GenerateMemories with direct_memories_source or direct_contents_source when consolidation is needed instead of manual create.
  • Set granular TTL values for different memory creation and update paths.
  • Use scopes such as user_id to isolate customer memories.
  • Use managed topic labels to filter customer data by category.
  • Use revision labels such as data_source and verified for governance workflows.

Gemini Data Analytics: A2A HTTP API Sample

  • Retrieve the Agent Card first to verify connectivity and agent capabilities.
  • Use uuid.uuid4() to create unique message_id values.
  • Use blocking=False for long-running tasks and poll terminal task states.
  • Use bounded backoff while polling task status.
  • Inspect both task artifacts and direct message content or metadata.

Gemini Data Analytics: A2A SDK API Sample

  • Use Google Cloud user authentication and refresh credentials before creating the client.
  • Pass the bearer token through an httpx.AsyncClient Authorization header.
  • Use the high-level a2a-sdk instead of manual stubs.
  • Enable streaming and polling in ClientConfig.
  • Use a UUID for each A2A message_id.

Intro to Gemini Data Analytics

  • Provide critical context for the agent through system_instruction.
  • Attach example_queries and glossary_terms when using BigQuery datasources.
  • Attach looker_golden_queries only when Looker is selected and the flag is enabled.
  • Use updateMask when patching data agent fields.
  • Route streamed system messages by type: text, schema, data, chart, and error.

Intro to Gemini Data Analytics

  • Provide business and data context in system_instruction to improve answer quality.
  • Use structured datasource references for BigQuery, Looker, or Looker Studio.
  • Add BigQuery example queries and glossary terms when available.
  • Use validated Looker golden queries when using Looker context.
  • Preserve existing IAM policy by getting it before setting updated permissions.

Managed Agents API - Analyzing the 2026 World Cup

  • Validate project API enablement and IAM roles before creating the agent.
  • Mask access tokens in logs instead of printing full secrets.
  • Refresh OAuth tokens when missing or expired.
  • Use timezone-aware UTC expiry checks for credentials.
  • Keep agent skills in a configured Cloud Storage bucket or use the Google Cloud Skills Registry.

Lyria 3 Music Generation

  • Set response_modalities to include both AUDIO and TEXT when both outputs are needed.
  • Provide genre, vocal style, and instruments in prompts for text-to-music generation.
  • Use types.Part.from_bytes with the correct image MIME type for image-conditioned generation.
  • Read PROJECT_ID and LOCATION from environment variables when notebook parameters are unset.
  • Clean model text output before displaying Markdown and audio.

Get started with Chirp 3: Instant custom voice

  • Use application default credentials and set the quota project before API calls.
  • Separate reference audio and voice talent consent audio in the generateVoiceCloningKey request.
  • Base64-encode WAV audio before sending it in the JSON payload.
  • Check for missing audioContent before trying to play generated audio.
  • Encapsulate REST calls and audio conversion in helper functions before wiring the UI.

Create a Multi-Speaker Podcast with Gemini 2.0 & Text-to-Speech

  • Use controlled generation with response_mime_type set to application/json and a response_schema.
  • Parse Gemini output with json.loads and handle JSONDecodeError and KeyError.
  • Skip audio synthesis when no dialogue is generated.
  • Use a regional Text-to-Speech API endpoint based on TTS_LOCATION.
  • Keep speaker labels constrained to R and S for MultiSpeakerMarkup turns.

Narrate a Multi-character Story with Gemini and Text-to-Speech

  • Use response_schema with Gemini to produce structured output for downstream processing.
  • Use a narrator voice for scene settings and title narration.
  • Filter available voices before assigning them to characters.
  • Use one Text-to-Speech call per dialogue line when different voices are required.
  • Combine generated clips with short silence between lines and remove intermediate MP3 files.

Combining Semantic & Keyword Search: A Hybrid Search Tutorial with Agent Platform Vector Search

  • Store both embedding and sparse_embedding on each item when building a hybrid index.
  • Fit the sparse vectorizer on the corpus before generating item and query sparse embeddings.
  • Use rrf_ranking_alpha to control the weight between dense and sparse search results.
  • Consider subword tokenizers, BM25, or SPLADE instead of basic word-level TF-IDF for production requirements.
  • Use fast Vector Search retrieval followed by reranking for higher-quality production retrieval or recommender systems.

Log Anomaly Detection & Investigation with Text Embeddings + BigQuery Vector Search

  • Aggregate daily user actions before embedding to make log analysis easier, faster, and more cost-effective.
  • Include principal, action, resource, container, channel, IP, and count in the text used for embeddings.
  • Normalize resource IDs and user-agent channels with BigQuery UDFs before embedding.
  • Remove exact duplicate matches when comparing suspicious actions to historical actions.
  • Use distance thresholds and visualizations to support anomaly investigation.

Anomaly Detection of Infrastructure Logs using Gemini and BigQuery Vector Search

  • Group raw logs by a meaningful session identifier before summarization.
  • Use a one-shot prompt and instruct Gemini not to repeat raw logs or speculate.
  • Persist intermediate outputs in BigQuery tables so long-running jobs can resume after failures.
  • Batch ML.GENERATE_TEXT calls to reduce loss from timeouts or quota exhaustion.
  • Use CLUSTERING task_type for embeddings when the downstream task is clustering-based anomaly detection.

Introduction to Agent Platform Vector Search 2.0

  • Use a schema-enforced collection with separate data_schema and vector_schema.
  • Use auto-embeddings by leaving vectors empty when vertex_embedding_config is configured.
  • Use random.seed(42) for reproducible sampling of the product dataset.
  • Use batch_create_data_objects instead of one create request per product for bulk imports.
  • Use task_type=“RETRIEVAL_DOCUMENT” for indexed product names and task_type=“QUESTION_ANSWERING” for semantic queries.

AG2 (formerly Autogen) Multi-Agents Example on Vertex AI Agent Engine

  • Authenticate in Colab with google.colab.auth or use Application Default Credentials outside Colab.
  • Test ResearchApp locally before deploying to Vertex AI Agent Engine.
  • Use Cache.disk() to reduce inference cost.
  • Use human_input_mode=“NEVER” for the deployed UserProxyAgent query flow.
  • Clean up deployed Agent Engines to avoid unnecessary costs.

Building and Deploying a Google Maps API Agent with Agent Engine

  • Test individual tool functions before wiring them into the agent.
  • Test the agent locally before deploying it to Agent Engine.
  • Use temperature 0 for deterministic agent behavior in this workflow.
  • Declare deployment requirements explicitly in the Agent Engine create config.
  • Grant only the needed Storage Object User role to the Agent Engine service account for bucket access.

Building a Conversational Search Agent with Agent Engine and RAG on Vertex AI Search

  • Test the Vertex AI Search function directly before wiring it into the agent.
  • Test the agent locally before deploying it to Agent Engine.
  • Use ChatMessageHistory and session_id to preserve conversational context across follow-up questions.
  • Set deployment requirements explicitly so Agent Engine has the needed LangChain and Discovery Engine packages.
  • Use temperature 0 for the demonstrated search agent behavior.

Semantic Router Agent

  • Use a structured Pydantic schema for router output instead of parsing free-form text.
  • Keep router temperature low and seed fixed for more deterministic classification.
  • Store user and model messages per turn so later calls can include conversation history.
  • Route unsupported inputs to a fallback response instead of an expert node.
  • Use LangGraph MemorySaver with thread_id to preserve session-specific state.

Migrate from PaLM to Gemini model

  • Use an EvalTask with multiple metrics when comparing foundation models.
  • Include prompt and reference columns in the summarization evaluation dataset.
  • Generate unique experiment run names with notebook_utils.generate_uuid(8).
  • Use at least 100 examples for best evaluation results.
  • Visualize qualitative metrics separately from ROUGE and BLEU.

Evaluate LangChain

  • Decompose multi-turn chats into per-turn examples with conversation_history before batch prediction.
  • Use temperature=0 for repeatable chatbot and evaluator outputs.
  • Map dataset columns explicitly with metric_column_mapping={“prompt”: “user”}.
  • Combine built-in metrics such as fluency, coherence, and safety with a task-specific custom metric.
  • Log evaluation results under an experiment name for run comparison.

Evaluating multimodal task

  • Use a structured response schema with score and explanation for autorater outputs.
  • Define a clear metric definition, criteria, rating rubric, and evaluation steps in the custom metric prompt.
  • Include both summary metrics and row-based metrics when reviewing evaluation results.
  • Inspect sampled explanations to validate the evaluation behavior.
  • Clean up the Vertex AI experiment when it is no longer needed.

Overview

  • Load API keys from environment variables and prompt with getpass when missing.
  • Use explicit task context to enforce sequential dependencies between agents.
  • Store test inputs and expected reference trajectories in a Phoenix dataset.
  • Evaluate trajectories with multiple metrics: exact match, precision, in-order match, and any-order match.
  • Add a custom code evaluator for agent-name order matching.

Intro to Function Calling with the Gemini API & Python SDK

  • Define clear functions with specific parameters and data types instead of parsing freeform text.
  • Set temperature=0 for functions that require deterministic parameter values.
  • Attach tools when creating the chat session to avoid sending them with every request.
  • Send external system results back with Part.from_function_response before asking Gemini for the final user-facing answer.
  • Handle multiple function calls in one turn by returning a list of function responses.

Real-time Retrieval Augmented Generation (RAG) using the Multimodal Live API with Gemini 2.0

  • Use genai.Client with vertexai=True, project, and location for Vertex AI endpoint calls.
  • Ground domain-specific answers with retrieved document context instead of relying on pretrained model knowledge.
  • Use the same embedding model for document chunks and user query embeddings.
  • Keep page number, chunk number, and document metadata with retrieved chunks.
  • Add exponential retry handling for quota-sensitive embedding and answer generation calls.

Nano Banana 🍌: Gemini 2.5 Flash Image Recipes

  • Defines shared model and generation configuration before running recipes.
  • Uses helper functions to create canvases and display generated images consistently.
  • Checks response candidates and inline image data before displaying output.
  • Uses blank canvases to guide aspect ratio-sensitive outputs.
  • Uses source and reference images with text prompts for grounded image edits.

From API to Report: Building a Currency Analysis Agent with LangGraph and Gemini

  • Separate the workflow into distinct nodes for API interaction, data validation, and report generation.
  • Use LangChain tools to expose external data sources to the agent.
  • Use conditional graph edges to route between tool execution and review.
  • Compile the graph with MemorySaver to support state management.
  • Use system and user prompts to define tool usage and the analysis task.

Chain of Thought & ReAct

  • Use one-shot exemplars to show the model the desired reasoning format.
  • Append “Let’s think step by step.” for zero-shot chain-of-thought reasoning.
  • Use self-consistency by generating multiple candidate answers and selecting the most popular result.
  • Avoid nested ReAct tool calls; parse work into separate actions.
  • Use external tools when the model lacks current or external information.

Text Summarization with Generative Models on Vertex AI

  • Initialize Vertex AI with an explicit project and location before model calls.
  • Use low temperature for concise summary generation when consistency is desired.
  • Adjust temperature, max_output_tokens, top_k, and top_p for different output styles.
  • Write prompts that specify the desired format, such as bullet points, TL;DR, to-dos, or title options.
  • Evaluate generated summaries against human-created summaries with ROUGE metrics.

Get Started with Gemini Preference Optimization

  • Validate chosen and rejected examples before writing tuning data.
  • Use JSONL with one Gemini preference example per line for training data.
  • Upload training and validation files to Cloud Storage before tuning.
  • Use a validation dataset with the tuning job.
  • Start with recommended hyperparameters and avoid beta=0 because it prevents learning.

Supervised Fine-tuning Gemini 2.5 Flash for Predictive Maintenance

  • Use train, validation, and test splits so evaluation samples are not seen during tuning.
  • Use Cloud Storage URIs for tuning datasets consumed by Vertex AI.
  • Set low temperature and max output tokens for classification-style tuned model evaluation.
  • Poll tuning job state until it leaves pending or running states before using the endpoint.
  • Check for empty data and failed job creation before launching or monitoring tuning.

Supervised Fine-tuning Gemini 2.5 Flash for Visual Defect Detection

  • Uses a separate train, validation, and test split with an 80/10/10 split.
  • Stores image files and JSONL tuning data in Cloud Storage for the fine-tuning service.
  • Uses guard clauses for missing data, unavailable endpoints, unexpected sample formats, and response parsing.
  • Sets a low temperature for deterministic classification during evaluation.
  • Uses a smaller adapter size for faster tuning in the demo.

AI-Assisted Data Science Workflows in BigQuery

  • Create a dedicated BigQuery dataset to contain tutorial tables, views, and models.
  • Use ML.DESCRIBE_DATA for quick exploratory statistics before modeling.
  • Filter raw data and add engineered features before AI enrichment and clustering.
  • Define an explicit AI.GENERATE_TABLE output schema for reliable typed columns.
  • Inspect enriched rows and generated embeddings before downstream modeling or indexing.

Analyzing movie posters in BigQuery with Gemini

  • Create a dedicated dataset for the demo resources.
  • Use a Cloud resource connection for BigQuery access to Cloud Storage objects.
  • Use output_schema with AI.GENERATE when extracting structured fields.
  • Store intermediate Gemini analysis and embedding results in BigQuery tables.
  • Use public datasets and joins to enrich generated analysis with structured metadata.

Semantic Analysis in BigQuery with AI Functions

  • Use managed AI functions for analysts who want prompt optimization handled by BigQuery.
  • Use AI.SCORE for semantic ranking by subjective criteria.
  • Use AI.CLASSIFY with explicit categories and a fallback category such as All Pets.
  • Use AI.IF for semantic filtering in WHERE clauses and semantic joins in JOIN ON clauses.
  • Use AI.GENERATE_BOOL for SELECT-clause enrichment when full prompt control is needed.

BigQuery DataFrames ML: Prescription Drug Name Generation

  • Use PROJECT_ID from user input or GOOGLE_CLOUD_PROJECT environment variable.
  • Create or reuse a BigQuery connection before invoking GeminiTextGenerator.
  • Filter out missing, duplicate, short, spaced, and generic-matching brand names before using examples.
  • Use random_state when sampling examples for reproducible few-shot prompts.
  • Limit batch rows for demonstration purposes.

Text + multimodal embedding generation and vector search in BigQuery

  • Create a dedicated BigQuery dataset before loading tutorial tables.
  • Use BigQuery remote models instead of exporting data to call Vertex AI separately.
  • Store embeddings in ARRAY columns for reuse with VECTOR_SEARCH.
  • Combine product_name and description before generating text embeddings to improve semantic context.
  • Order vector search results by distance ascending.

Introduction to Generative AI functions in BigQuery

  • Create a Cloud resource connection for BigQuery access to Vertex AI services.
  • Grant the connection service account the required BigQuery connection and Vertex AI roles.
  • Wait for IAM changes to propagate before running model calls.
  • Use model_params to control temperature, maxOutputTokens, and thinking_budget.
  • Use output_schema when generated results need typed structure.

Test Document AI Gemini

  • Process the same input document through both extraction paths before comparing outputs.
  • Use separate prompts for extraction and comparison.
  • Upload the document to temporary Cloud Storage for model-based extraction, then delete the temporary file.
  • Keep Document AI configuration values explicit: project, location, processor, file path, and MIME type.

Analyze Multimodal Data in BigQuery

  • Use ObjectRefs as secure pointers instead of storing unstructured file bytes in BigQuery tables.
  • Use object tables when every file in a GCS path should automatically become an ObjectRef row.
  • Use OBJ.MAKE_REF and OBJ.FETCH_METADATA when URI columns already exist in structured tables.
  • Join structured records with ObjectRef columns to create a reusable multimodal table.
  • Define output_schema for AI.GENERATE and AI.GENERATE_TABLE to return typed, queryable results.

Performing Semantic Search in BigQuery

  • Create a BigQuery Cloud resource connection before defining the remote model.
  • Grant Vertex AI User to the connection service account used by BigQuery.
  • Use SEMANTIC_SIMILARITY as the embedding task type for search embeddings.
  • Concatenate question title and body before embedding to include both fields.
  • Create an IVF cosine vector index on ml_generate_embedding_result before VECTOR_SEARCH.

Analyze a codebase with Gemini in Vertex AI

  • Use a shallow clone when loading the GitHub repository.
  • Create a context cache for the codebase to avoid resending the full context on every request.
  • Include a system instruction that frames the model as a coding expert for code-related questions.
  • Use streaming generation for longer developer guides, troubleshooting guides, and recommendations.
  • Separate tool-based GitHub issue retrieval from cached-content prompts because tools cannot be added at runtime with cached content.

Use Retrieval Augmented Generation (RAG) with Gemini API

  • Use retrieved repository context rather than relying only on zero-shot prompting for code generation.
  • Split Python code with RecursiveCharacterTextSplitter.from_language using chunk overlap.
  • Persist crawled GitHub file URLs to avoid repeatedly downloading the file list.
  • Use low temperature for concise code generation.
  • Return source documents from RetrievalQA for traceability.

Code Vulnerability Scanning & Automated Remediation using Gemini API in Vertex AI (Gemini 2.0)

  • Add each filename as a separator before its code so the model can identify findings per file.
  • Use a clear prompt that specifies vulnerability name, description, recommendations, and recommended code.
  • Set generation parameters including temperature, top_p, top_k, candidate_count, and max_output_tokens.
  • Export findings to CSV and JSON for further analysis, benchmarking, and integration with security tools.
  • Treat the workflow as experimental and continue validation before using it as a robust security tool.

Reduce Tech Debt with Gemini 3 Pro

  • Generate passing baseline tests before refactoring legacy code.
  • Verify generated tests against the original code before generating replacement code.
  • Create a detailed specification from the legacy source before creating a design document.
  • Generate new code from the specification and design document, not directly from the legacy code.
  • Run the same tests against the generated code for final verification.

Comparing LlamaIndex and LlamaParse for Dense Document Questioning Answering on Vertex AI

  • Compare multiple parsing approaches against the same source document and query set.
  • Use similarity_top_k=2 consistently across query engines for apples-to-apples comparison.
  • Use domain-specific parsing instructions for LlamaParse on 10-Q financial documents.
  • Extract question and keyword metadata from parsed nodes before embedding for richer retrieval.
  • Print response source nodes, relevance scores, file names, page labels, and file paths for answer inspection.

Document Processing with Gemini

  • Use explicit system instructions for extraction, classification, QA, and summarization tasks.
  • Use response_schema and response_mime_type to enforce structured JSON or enum outputs.
  • Use Pydantic models to define extraction schemas with field descriptions.
  • Use Part.from_uri for GCS or HTTPS PDFs and Part.from_bytes for local PDF bytes.
  • Use response.parsed when consuming structured model outputs.

Patents Document Understanding with Gemini

  • Use a detailed prompt plus responseMimeType application/json and responseSchema for controlled generation.
  • Include the PDF as fileData with mimeType application/pdf instead of extracting document text separately.
  • Use a systemInstruction to set the model role as an expert at analyzing patent documents.
  • Write batch requests to BigQuery and let the batch job return outputs to a BigQuery table.
  • Poll batch job state before reading destination results.

Sheet Music Analysis with Gemini

  • Use environment variables for GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION when notebook parameters are not provided.
  • Set safety settings and a domain-specific system instruction before generation.
  • Use Part.from_uri with explicit MIME types for PDF and audio inputs.
  • Use response_mime_type=“application/json” before parsing model output with json.loads.
  • Copy PDF pages into a writer before adding metadata and saving the file.

Text Summarization of Large Documents using LangChain 🦜🔗

  • Use environment variables as fallbacks for Google Cloud project and region configuration.
  • Keep summarization instructions in PromptTemplate objects for stuffing, map, combine, question, and refine prompts.
  • Test stuffing on a small page subset and handle exceptions from oversized context.
  • Use map_reduce or refine when document length exceeds what stuffing can handle.
  • Set return_intermediate_steps=True to inspect chunk-level outputs.

Automating Income Taxes with Gemini

  • Use temperature=0 for deterministic classification and extraction.
  • Constrain classification with an Enum response schema.
  • Use document-specific Pydantic schemas for typed structured extraction.
  • Map classified document types to their extraction schemas before calling the model.
  • Normalize parsed JSON before joining it back into the source DataFrame.

Using Gemini in Education

  • Use a low default temperature for more consistent responses.
  • Use few-shot examples to guide response structure and formatting for text correction.
  • Ask for step-by-step reasoning to reduce hallucinations in math tasks.
  • Use structured output requests such as JSON lists or tables when asking detailed questions.
  • Ask video questions using the video only and request timestamps plus source type such as image, text, or speech.

Evaluate

  • Use environment variables for project, location, and image path configuration.
  • Set RANDOM_STATE for reproducible sampling.
  • Use stratify=True for class-balanced evaluation samples.
  • Evaluate both all classes and a targeted subset of high-interest classes.
  • Inspect summary metrics together with confusion matrices and detailed rows.

ReAct (Reasoning + Acting) + Custom tool for Healthcare NL API + Gemini 2.0 + LangChain

  • Use Healthcare NLP to preprocess unstructured medical records before asking the LLM for code suggestions.
  • Limit extracted tool terms to clinically relevant entity categories before passing them to the agent.
  • Use temperature 0 for more deterministic medical coding output.
  • Hide ReAct reasoning steps with verbose=False when reasoning traces should not be shown.
  • Use self-consistency by running repeated attempts and selecting the majority response.

Building Knowledge Graphs with Gemini

  • Use only the provided input data to avoid relying on memorized general knowledge.
  • Use explicit domain terminology such as entities, relationships, nodes, and edges.
  • Include deterministic settings such as temperature=0.0, top_p=0.0, and seed=42 for extraction tasks.
  • Separate data schema, extraction instructions, and output format in the prompt.
  • Use TSV for table-shaped outputs when token efficiency matters.

Know Your Customer Use Case - Gemini Grounding with Google Search

  • Use explicit system instructions to define role, scope, output format, and no-result behavior.
  • Ground sensitive claims with Google Search and display source metadata for verification.
  • Check for missing candidates, grounding metadata, grounding supports, and out-of-bounds chunks before extracting sources.
  • Separate reusable single-entity generation from batch entity processing.
  • Evaluate outputs with task-specific custom metrics and rubrics.

Creative Content Generation with Gemini in Vertex AI and Imagen

  • Add product description to improve generated output quality.
  • Use product images as multimodal context for marketing messages.
  • Target generated posts to specific platforms such as Facebook, Instagram, LinkedIn, and Twitter.
  • Reuse generated posts as reference context when personalizing for audience segments.
  • Prepare an expanded base image and mask before calling Imagen outpainting.

Video Data Curation - Video Quality Filtering

  • Discard low-quality clips to use limited modeling compute efficiently.
  • Use metadata filters such as FPS, duration, resolution, brightness, and aspect ratio for downstream modeling requirements.
  • Use structured JSON output for separating detected text from watermark descriptions.
  • Set Gemini temperature to 0.0 for deterministic extraction-style calls.
  • Manually inspect low-scoring videos to choose an aesthetic score threshold.

Data Curation Pipeline: Splitting and Transcoding

  • Store videos in Cloud Storage before calling the Multimodal Embeddings API.
  • Use a semaphore with ThreadPoolExecutor to control embedding API concurrency.
  • Capture errors per video URI instead of failing the whole embedding run.
  • Store embeddings in BigQuery for reproducible vector search workflows.
  • Create a BigQuery VECTOR INDEX with COSINE distance before large vector search workloads.

Data Curation Pipeline: Splitting and Transcoding

  • Filter videos by duration and resolution before downstream processing.
  • Use FFmpeg directly via subprocess for robust fixed-length splitting with stream copy.
  • Clean up temporary files and directories after local video processing.
  • Sort segment files before batch transcoding for consistent processing order.
  • Test transcoding parameters on a sample of video data before applying them broadly.

Multimodal Sentiment Analysis with Gemini

  • Set project and location before creating the Gen AI client.
  • Use Part.from_uri with an explicit MIME type for Cloud Storage audio inputs.
  • Compare direct audio analysis with transcript-based analysis to understand modality differences.
  • Use speaker labels in prompts for conversation analysis.
  • Set response_modalities to TEXT when requesting text-only output.

Productivity Coaching with Gemini and Google Calendar

  • Use system instructions to define Gemini’s coaching role and analysis criteria.
  • Start with sample calendar screenshots before connecting live user data.
  • Request only readonly Calendar API access for event analysis.
  • Validate tool input bounds before calling the Calendar API.
  • Return a concise subset of event fields: summary, start, end, and status.

Slide Generation with Gemini and Marp

  • Use PROJECT_ID from GOOGLE_CLOUD_PROJECT when the placeholder is not replaced.
  • Set a system instruction to restrict Gemini output to Markdown.
  • Provide the Marp repository as context so generated slides can follow Marp style and formatting.
  • Use max_output_tokens=8192 to allow enough room for a complete slide deck.
  • Write generated Markdown to slide-deck.md before rendering it with Marp CLI.

Multimodal retail recommendation: using Gemini to recommend items based on images and image reasoning

  • Combine text and image in a single prompt for visual understanding.
  • Label each candidate image within the prompt so the model can reference options clearly.
  • Provide the available item images when recommendations should be limited to store inventory.
  • Use response_mime_type=“application/json” when recommendations need to plug into an application.

Retail AI Location Strategy: Autonomous Site Selection & Market Analysis

  • Use Search Grounding for fresh demographics, growth, rental, and market viability data.
  • Wrap Google Maps Places API as a tool for real competitor names, locations, and ratings.
  • Base gap analysis on prior market research and competitor findings rather than isolated prompts.
  • Use code execution for density, saturation, and scoring calculations.
  • Use Pydantic schemas and response_schema to make final recommendations machine-readable.

RAG Based on Sensitive Data Protection using Faker

  • Anonymize source documents before creating embeddings.
  • Anonymize the user query before retrieval against anonymized embeddings.
  • De-anonymize the generated response only after the RAG chain returns an answer.
  • Reuse existing mappings when the same original data is detected again.
  • Constrain the prompt to say it does not know when context is insufficient.

Intra Knowledge QnA

  • Initialize Vertex AI with an explicit project and location.
  • Use chunk overlap to preserve context across document splits.
  • Persist the vector database to disk for reuse.
  • Limit retrieval to the top 3 similar documents for answer generation.
  • Print unique source document paths from retrieved context.

Retrieval Augmented Generation(RAG) with AlloyDB

  • Pin package versions for reproducible notebook setup.
  • Initialize Vertex AI with explicit project and location.
  • Use the AlloyDB connector and SQLAlchemy engine instead of raw unmanaged connections.
  • Create the google_ml_integration and vector extensions before generating and querying embeddings.
  • Concatenate title and abstract before embedding to improve retrieval context.

Building a Gen AI RAG application with Vertex AI Feature Store and BigQuery

  • Use BigQueryVectorStore for prototyping because it requires no infrastructure startup time.
  • Split documents so a few chunks can fit within the LLM context length.
  • Store document source, document name, and chunk number in metadata.
  • Use add_texts_with_embeddings when embeddings are already precomputed.
  • Use VertexFSVectorStore for production-ready user-facing Gen AI applications that need low-latency retrieval.

Augment Gemini Output with Vector Embeddings from BigQuery

  • Create BigQuery remote models through a Cloud resource connection for Vertex AI access.
  • Filter out rows with empty abstract and title before generating embeddings.
  • Store generated embeddings in a BigQuery table for reuse.
  • Check INFORMATION_SCHEMA.VECTOR_INDEXES for coverage_percentage and last_refresh_time before relying on the index.
  • Use the same embedding model for indexed embeddings and query embeddings.

Run RAG Pipelines in BigQuery with BQML and Vector Search

  • Use a Cloud resource connection so BigQuery can call Cloud Storage, Vertex AI, and Document AI.
  • Grant required IAM roles to the BigQuery connection service account before creating remote models.
  • Store source PDFs in Cloud Storage and expose them through a BigQuery object table.
  • Parse Document AI JSON into chunk content and metadata before embedding.
  • Use ML.GENERATE_EMBEDDING for both document chunks and user queries to keep embeddings consistent.

Spatial understanding with Gemini 3

  • Use response_mime_type=“application/json” and response_schema=list[BoundingBox] for controlled generation.
  • Put repeated output-format rules in system_instruction to keep prompts shorter.
  • Use safety_settings with HARM_CATEGORY_DANGEROUS_CONTENT set to BLOCK_ONLY_HIGH.
  • Give repeated objects unique labels based on distinct characteristics such as color, size, or position.
  • Scale normalized box coordinates by image width and height before drawing overlays.

Identifcation of Scene Transitions in Movies Using Gemini

  • Use temperature 0 for consistent scene boundary extraction.
  • Provide both video and VTT inputs to combine visual and dialogue cues.
  • Use response_mime_type application/json with response_schema for structured outputs.
  • Define explicit criteria for narrative, visual, dialogue, audio, and cohesion signals.
  • Do not count jump-cuts or insert shots as transitions unless they indicate meaningful narrative shifts.

YouTube Video Analysis with Gemini

  • Use Part.from_uri with mime_type=“video/webm” for public YouTube video inputs.
  • Use response_mime_type=“application/json” and response_schema for structured extraction.
  • Constrain enum fields in the response schema when only specific values are valid.
  • Prompt the model to use only information in the video itself for extraction.
  • Use asynchronous generation for analyzing multiple videos more efficiently.

Enhanced Vision Assistant with Gemini

  • Use environment variables for GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION when possible.
  • Allow an explicit credentials path but also support application default credentials.
  • Filter small detections with MIN_OBJECT_SIZE before generating guidance.
  • Prioritize navigation guidance by urgency using NavigationPriority.
  • Clean up camera/audio resources and temporary audio files in stop().

Qwen 3 evaluation - Bring your own data eval

  • Use a custom dataset for domain-specific model assessment.
  • Compare a candidate response against a baseline_model_response row by row.
  • Use pairwise_summarization_quality for summary quality comparison.
  • Log evaluation runs to a Vertex AI Experiment for tracking and comparison.
  • Optionally clean up experiments after running evaluations.

Hugging Face DLCs: Using Gemma for running evaluations with Vertex AI Gen AI Evaluation

  • Initialize Vertex AI with project and location before creating models, endpoints, and evaluations.
  • Use a tokenizer chat template before sending prompts to the Hugging Face TGI endpoint.
  • Wrap endpoint prediction in a model function so EvalTask can call it consistently.
  • Track evaluation runs with Vertex AI Experiments through experiment and experiment_run_name.
  • Combine reference-based metrics such as rouge_l_sum with model-based metrics such as summarization_quality and fluency.

Deploying Multiple LoRA Adapters on Vertex AI with vLLM

  • Store model files in Cloud Storage instead of baking them into a large Docker image.
  • Enable parallel composite uploads for faster Cloud Storage transfers.
  • Use a startup entrypoint that fails fast when GCS downloads fail before starting vLLM.
  • Set temperature to 0.0 for deterministic SQL generation.
  • Configure Vertex AI health and predict routes explicitly for the serving container.

Use Any OSS Gen AI Model Against Your BigQuery Data

  • Check accelerator quota before deploying the model endpoint.
  • Use a dedicated Cloud Storage bucket or create a temporary regional bucket for experiment artifacts.
  • Grant only the needed service accounts access to Cloud Storage and Vertex AI.
  • Use low temperature and constrained output tokens for structured extraction.
  • Persist generated outputs to a BigQuery table before downstream analytics.

Deploying Llama 3 on Google Kubernetes Engine with Cloud Functions and vLLM

  • Uses private GKE nodes and an internal load balancer for the model service.
  • Separates the model server from the API layer by putting Cloud Functions in front of GKE.
  • Uses a VPC connector so the Cloud Function can reach the private GKE endpoint.
  • Passes the internal endpoint IP through an environment variable instead of hardcoding it in function code.
  • Creates a dedicated service account for the Cloud Function and grants required IAM roles.

Create a Vertex AI Datastore and Search Engine

  • Use ClientOptions with a regional discoveryengine endpoint when location is not global.
  • Use the default_collection parent path for datastore and engine creation.
  • Import documents into the default_branch from Cloud Storage using data_schema=“content”.
  • Use INCREMENTAL reconciliation mode for document import.
  • Enable SEARCH_TIER_ENTERPRISE and SEARCH_ADD_ON_LLM when summaries or advanced LLM features are needed.

Defining custom attributes based on URL patterns in Vertex AI Search Website Datastores

  • Use global datastore location unless there is a specific reason to use us or eu.
  • Verify schema and URL mapping with getUriPatternDocumentData after setting it.
  • Make custom attributes indexable, retrievable, and searchable when they should affect retrieval and appear in responses.
  • Poll datastore and app availability because creation can take a few minutes.
  • Delete the app and datastore when no longer needed.

Event-based Triggering of Manual Recrawl for Vertex AI Search Advanced Website Datastores

  • Use a Cloud Storage staging bucket for JSON files containing URLs to recrawl.
  • Trigger processing on google.cloud.storage.object.v1.finalized events and filter for .json files.
  • Use functions-framework, described as the recommended way at the time of the notebook.
  • Fetch an access token with google-auth default credentials before calling the REST API.
  • Set a request timeout and retry connection or timeout failures up to three times.

Parsing and Chunking in Vertex AI Search: Featuring BYO Capabilities

  • Create the datastore with layout parser for complex documents containing tables and lists.
  • Include ancestor headings with chunks to preserve heading context.
  • Use incremental reconciliation for document and chunk imports.
  • Poll datastore creation and import long-running operations before retrieving processed documents.
  • Export chunked JSON to Cloud Storage for offline review or editing before BYOC import.

Query-Level Boosting, Filtering, and Facets for Vertex AI Search Website Datastores

  • Evaluate Vertex AI Search results without extra rules before incrementally adding custom filters or boosts.
  • Apply boosting and filtering during retrieval rather than post-processing search results.
  • Use metadata fields as schema hooks for filters, facets, and boost conditions.
  • Use facets to translate user preferences into subsequent filtered search requests.
  • Start with smaller boost values and adjust as needed.

Recording Real-Time User Events in Vertex AI Search Datastores

  • Report both search events and view-item events with the same attribution token.
  • Include the impressions shown to the user in the search event.
  • Use userPseudoId to associate events with a user without requiring a named account.
  • Use JavaScript Pixel as the recommended alternative when the customer can control the page source.
  • Purge synthetic notebook events after testing.

Q&A Chatbot with Vertex AI Search for summarized website results without advanced indexing

  • Initialize Vertex AI with project and location before model use.
  • Use query expansion and spell correction with AUTO mode in SearchRequest.
  • Request snippets with ContentSearchSpec.SnippetSpec(return_snippet=True).
  • Handle empty search responses and exceptions by returning None or a fallback message.
  • Use low temperature for concise grounded answer generation.

Virtual Try-On: Batch Generation Pipeline

  • Reads project and region from environment variables when notebook parameters are not set.
  • Displays input images before generation and displays generated images afterward for visual inspection.
  • Uses response.raise_for_status() when fetching public image URLs.
  • Tracks current job and total jobs while processing the Cartesian product of inputs.
  • Catches per-job exceptions and skips failed jobs instead of stopping the whole batch.

Back to Applied Use Cases · Best Practices Map

Graph View

Backlinks