Getting Started — Best Practices
Distilled from 53 notebooks tagged Getting Started in the GoogleCloudPlatform/generative-ai repository. The From the notebooks section below cites the per-notebook source for grounding.
Do this
- Use a real Google Cloud project with the required APIs enabled before running examples; set PROJECT_ID and LOCATION explicitly or through GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION fallbacks.
- Prefer the Google Gen AI SDK unified interface for Vertex AI projects, and use REST, cURL, OpenAI-compatible Chat Completions, LangChain, or WebSockets when the notebook’s integration path requires that level of control.
- Keep GenerateContentConfig or equivalent request configuration explicit: system_instruction, safety_settings, generation parameters, thinking configuration, response_modalities, response_mime_type, response_schema, tools, and media_resolution should be deliberate.
- Use concise, specific prompts that ask one task at a time; add representative zero-shot, one-shot, or few-shot examples when needed, and state when provided context is the only source of truth.
- Use system instructions to steer behavior, formatting, persona, and task boundaries, but keep persona constraints controlled because they can override other instructions.
- Use streaming for user-facing latency, async calls for higher-throughput applications, batch inference for large non-latency-sensitive workloads, and Live API or WebSockets for bidirectional audio, video, or native audio interaction.
- Inspect finish_reason, safety_ratings, usage_metadata, thoughts token counts, total token counts, and response parts before assuming text, image, audio, tool calls, or structured data are present.
- Use count_tokens or compute_tokens before dispatching large multimodal, long-context, video, audio, or document requests when token volume, latency, or cost matters.
- For structured output, set response_mime_type to application/json, provide response_schema or response_json_schema, mark required fields explicitly, use nullable fields when context may be missing, and consume response.parsed when available.
- For function calling, define clear OpenAPI-style function parameters, use deterministic settings where appropriate, execute tools in trusted application code, return function responses to the model, and handle multiple function calls in one turn.
- For Gemini 3 manual tool histories, preserve the full previous candidate content, including thought signatures, or use SDK chat history and automatic function calling so the SDK manages that state.
- For multimodal work, use Part.from_uri for Cloud Storage, public web, YouTube, PDF, audio, HTML, or video inputs and Part.from_bytes for local files; always provide correct MIME types and set response modalities to match expected outputs.
- For image and audio generation, check finish_reason before assuming output exists, handle text and inline_data parts separately, use ImageConfig for aspect ratio or size controls, and decode streamed PCM or base64 audio according to the notebook’s sample rate and format.
- For embeddings, RAG, and Vector Search, batch requests with pacing, match index dimensions to embedding output, normalize vectors when the notebook does so, configure chunk size and overlap, inspect retrieved context directly, and tune top_k or distance thresholds.
- For data agents, logging, evaluation, optimization, and IAM operations, preserve existing policies or configs before updating, use update masks where shown, provide datasource context such as glossary terms and example queries, and validate jobs or configs before starting long-running work.
Avoid this
- Running notebooks before enabling the required APIs, setting PROJECT_ID and LOCATION, authenticating in Colab, or configuring Application Default Credentials and quota project for local or cURL workflows.
- Using the wrong location or endpoint: some examples default to global, some require us-central1 or regional endpoints, context caching is not available on the global endpoint, and some BigQuery, TTS, STT, Live API, Vector Search, and batch workflows have regional constraints.
- Assuming every response contains plain text: safety blocking can make response.text None, image generation can return a non-STOP finish reason, message:send can return either a task or direct message, and generated content can appear across text, inline_data, thought, executable_code, code_execution_result, artifact, or metadata parts.
- Combining Gemini 3 thinking_level with legacy thinking_budget, trying to turn off thinking where the notebook says it cannot be turned off, or dropping thought_signature fields in raw manual function-calling histories, all of which can cause 400 errors.
- Expecting Gemini function calling to execute tools automatically; the notebooks keep tool execution in application code and require returned tool results to be appended back into the conversation.
- Mishandling multimodal or audio payloads: missing MIME types, incorrect base64 flags, non-public web URIs, wrong Cloud Storage project access, wrong PCM sample rate or endian format, or missing IMAGE or AUDIO in response_modalities.
- Failing to poll or bound long-running work such as batch jobs, Vector Search indexes, imports, data analytics tasks, prompt optimization, or logging propagation; many examples require result polling, backoff, timeouts, or cancellation.
- Overwriting IAM policies or leaving billable tutorial resources running; Set IAM Policy can replace existing permissions, and indexes, endpoints, collections, buckets, logging configs, batch jobs, and Workbench resources can continue to incur cost.
From the notebooks
- Create a baseline with default managed topics before adding custom topics.
- Use the same conversation for default and custom engines to make the comparison apples-to-apples.
- Keep custom topics focused on separate domain dimensions to reduce overlap and improve precision.
- Write detailed topic descriptions with examples and exclusions for what belongs in other topics.
- Use realistic few-shot conversations that show the desired memory extraction style.
Gemini Data Analytics: A2A HTTP API Sample
- Retrieve the Agent Card first to verify connectivity and agent capabilities.
- Use uuid.uuid4() to create unique message_id values.
- Use blocking=False for long-running tasks and poll terminal task states.
- Use bounded backoff while polling task status.
- Inspect both task artifacts and direct message content or metadata.
Intro to Gemini Data Analytics
- Provide critical context for the agent through system_instruction.
- Attach example_queries and glossary_terms when using BigQuery datasources.
- Attach looker_golden_queries only when Looker is selected and the flag is enabled.
- Use updateMask when patching data agent fields.
- Route streamed system messages by type: text, schema, data, chart, and error.
- Use a mandatory SKILL.md file with YAML frontmatter and markdown instructions.
- Make the skill name a unique identifier matching the skill package name.
- Start the skill description in third person as a capability statement.
- Use local_path so the SDK packages, compresses, uploads, provisions, and indexes the skill.
- Use timestamped skill IDs to avoid collisions during registration.
- Set response_modalities to include both AUDIO and TEXT when both outputs are needed.
- Provide genre, vocal style, and instruments in prompts for text-to-music generation.
- Use types.Part.from_bytes with the correct image MIME type for image-conditioned generation.
- Read PROJECT_ID and LOCATION from environment variables when notebook parameters are unset.
- Clean model text output before displaying Markdown and audio.
Gemini 3.1 Flash Text-to-Speech Generation
- Use VoiceConfig with PrebuiltVoiceConfig to choose documented voice options.
- Use response_modalities=[“AUDIO”] when requesting speech output.
- Use MultiSpeakerVoiceConfig with one SpeakerVoiceConfig per speaker for multi-speaker clips.
- Place audio tags immediately before the phrase or sentence they should influence.
- Avoid overusing audio tags and match them to natural changes in tone or pace.
Get started with Chirp 3 HD voices using Text-to-Speech
- Use PROJECT_ID from GOOGLE_CLOUD_PROJECT when no explicit project ID is provided.
- Build API_ENDPOINT from TTS_LOCATION so global and regional endpoints are handled consistently.
- Use ClientOptions to pass the Text-to-Speech API endpoint explicitly.
- Split long text into sentence chunks before streaming synthesis.
- Send the streaming configuration before streaming text inputs.
Get started with Chirp 3 Transcription
- Use AutoDetectDecodingConfig so the API detects audio encoding.
- Use a regional Speech-to-Text endpoint based on STT_LOCATION.
- Use Cloud Storage URIs for batch recognition inputs and outputs.
- Use language_codes=[“auto”] for language-agnostic transcription.
- Configure SpeakerDiarizationConfig inside RecognitionFeatures for diarization.
Get started with Gemini-TTS voices using Text-to-Speech
- Choose Cloud Text-to-Speech API when specific output encodings or bidirectional streaming are needed.
- Choose Agent Platform API when already using Gemini-TTS from AI Studio or other Agent Platform models.
- Use natural-language prompts to steer style, accent, pace, tone, and emotional expression.
- Use multi_speaker_voice_config to assign speaker aliases to specific voices.
- Stream audio chunks immediately to the frontend in real-time applications.
Getting Started with Text Embeddings + Agent Platform Vector Search
- Limit tutorial data size from the 23 million row Stack Overflow table before loading into memory.
- Batch embedding requests and add delay to avoid quota errors.
- Use dot product for similarity with the Google embedding model shown.
- Store Vector Search input as JSONL with id and embedding fields.
- Match index dimensions to the embedding output dimensionality.
Vector Search 2.0 Public Preview Quickstart
- Define data_schema and vector_schema before creating data objects.
- Normalize generated dense vectors before storing or searching them.
- Sort sparse embedding indices when generating sparse vectors.
- Use batch create, batch update, batch delete, and batch search for multi-object operations.
- Use output_fields to control returned data, vector, and metadata fields.
Intro to Batch Inference with the Gemini API
- Use batch inference for large input sets that are not latency sensitive.
- Specify explicit Cloud Storage or BigQuery output locations for completed predictions.
- Check batch job status with client.batches.get before retrieving results.
- List jobs with client.batches.list when inspecting project batch activity.
- Delete created batch prediction jobs during cleanup.
Using OpenAI libraries with Gemini on Vertex AI
- Use google-auth cloud-platform scoped credentials instead of static API keys.
- Use a regional Vertex AI endpoint when using context caching.
- Use tool_choice to control whether the model may call tools.
- Use response schemas with Pydantic when structured output is required.
- Use context caching for repeated large inputs to reduce tokens sent and lower request cost.
Intro to Generating and Executing Python Code with Gemini 3
- Use the Google Gen AI SDK unified interface with Vertex AI for enterprise-ready projects.
- Pass code_execution as a Tool instead of expecting code execution by default.
- Inspect executable_code and code_execution_result response parts separately.
- Save or display the output code, result, or outcome downstream in the application.
- Use chat sessions when code needs to be rewritten iteratively with history.
Intro to Structured Output with the Gemini API
- Set response_mime_type to application/json for JSON outputs.
- Use response_schema with Pydantic or OpenAPI-style schemas for controlled JSON generation.
- Use response_json_schema when JSON Schema conditional logic is needed.
- Use response.parsed to consume structured outputs as objects or dictionaries.
- Use text/x.enum when the model must choose from predefined enum values.
Migrate from PaLM to Gemini model
- Use an EvalTask with multiple metrics when comparing foundation models.
- Include prompt and reference columns in the summarization evaluation dataset.
- Generate unique experiment run names with notebook_utils.generate_uuid(8).
- Use at least 100 examples for best evaluation results.
- Visualize qualitative metrics separately from ROUGE and BLEU.
Intro to Function Calling with the Gemini API & Python SDK
- Define clear functions with specific parameters and data types instead of parsing freeform text.
- Set temperature=0 for functions that require deterministic parameter values.
- Attach tools when creating the chat session to avoid sending them with every request.
- Send external system results back with Part.from_function_response before asking Gemini for the final user-facing answer.
- Handle multiple function calls in one turn by returning a list of function responses.
- Use a Google Cloud project with Vertex AI API enabled for most users.
- Use streaming generation when responses should appear as they are generated.
- Set thinking_budget based on task complexity to manage quality and speed.
- Set thinking_budget to 0 for simpler examples where lower latency is desired.
- Inspect usage_metadata, thoughts_token_count, and total_token_count when evaluating thinking cost.
Intro to Gemini 2.5 Flash-Lite
- Use GenerateContentConfig to keep model parameters, system instruction, safety settings, tools, and schemas explicit per request.
- Use system_instruction to provide task context, persona, formatting rules, and interaction guidelines.
- Use safety_settings to adjust blocking behavior for specific harm categories.
- Inspect usage_metadata to see prompt, candidate, thoughts, and total token counts.
- Use response_schema with response_mime_type=“application/json” for controlled generation.
Gemini 2.5 Flash Image (Nano Banana 🍌) Generation
- Set response_modalities to match the desired output modalities.
- Use ImageConfig to specify aspect_ratio for generated images.
- Check finish_reason before assuming an image was generated.
- Iterate over response content parts and handle both text and inline_data.
- Use Part.from_bytes for local image inputs.
- Use a Google Cloud Project for authentication for most users.
- Set thinking_budget to control quality and speed of response.
- Inspect usage_metadata token counts when evaluating thinking budget behavior.
- Use include_thoughts only when summarized thoughts are needed.
- Use lower thinking budget for examples that do not need extra reasoning to reduce latency.
Gemini 3.1 Flash Image (Nano Banana 2 🍌) Generation
- Use types.GenerateContentConfig to set response_modalities, image_config, thinking_config, and tools.
- Check response.candidates[0].finish_reason before assuming an image was generated.
- Skip parts marked thought when displaying generated images unless inspecting thoughts intentionally.
- Use Part.from_bytes for local image or video bytes and Part.from_uri for YouTube or Cloud Storage inputs.
- Use Google Search tools when prompts need current web or image grounding.
Intro to Gemini 3.1 Flash-Lite
- Use thinking_level MINIMAL for low-complexity, lower-latency responses.
- Use higher thinking levels for tasks requiring deeper reasoning.
- Use Google Gen AI SDK chat history or append the full model response so thought signatures are handled automatically.
- Append response.candidates[0].content when doing manual function calling across turns.
- Set media_resolution based on detail needs and input length.
- Use Google Gen AI SDK chat history or append the full model response so thought signatures are handled automatically.
- Set thinking_level to low for simple instruction following or chat when lower latency is desired.
- Use high thinking level for non-trivial reasoning tasks such as concurrency-safe code generation.
- Set media_resolution per media part when different images or videos need different detail levels.
- Lower media resolution for very long inputs to manage token count.
- Use streaming for user-facing or chat-based interfaces to receive chunks immediately.
- Use async generation to improve throughput in web apps and multi-agent systems.
- Use per-part media resolution when different media inputs need different fidelity levels.
- Use automatic function calling when possible because the SDK manages thought signatures behind the scenes.
- Preserve response.candidates[0].content in manual tool histories to maintain reasoning context.
Intro to Gemini 3 Flash (Preview)
- Use the Google Gen AI SDK chat history or append the full model response so thought signatures are handled automatically.
- Set thinking_level to MINIMAL or LOW for lower-complexity tasks where lower latency is preferred.
- Use HIGH thinking_level for non-trivial reasoning tasks such as the double-checked locking C++ example.
- Lower media_resolution for long videos or extensive documents to fit token limits.
- Keep Gemini 3 temperature at its default value of 1.0 as recommended in the notebook.
Gemini 3 Pro Image (Nano Banana Pro 🍌) Generation
- Install or upgrade google-genai before running the notebook.
- Fallback to the GOOGLE_CLOUD_PROJECT environment variable when PROJECT_ID is not set.
- Use GenerateContentConfig with response_modalities and ImageConfig for image output control.
- Check finish_reason before assuming an image was generated.
- Skip thought parts when displaying final generated images.
Getting Started with Chat with Gemini
- Use environment variables for project and region fallback.
- Set system instructions when creating GenAI SDK chat sessions.
- Inspect response metadata including safety_ratings and usage_metadata.
- Retrieve chat history when validating stateful behavior.
- Add prior chat history as alternating user and model messages.
- Set GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION before building the API endpoint.
- Use the global aiplatform.googleapis.com host unless a regional location is configured.
- Set generation parameters such as temperature, top_p, top_k, max_output_tokens, candidate_count, and stop_sequences deliberately.
- Use safety_settings to configure harm blocking thresholds.
- Specify role values for multi-turn chat content.
Getting started with Gemini using Vertex AI in Express Mode
- Use streaming responses when reducing perceived latency matters.
- Use system instructions to steer model behavior for a task.
- Inspect finish_reason and safety_ratings when using safety settings.
- Use response_schema and response_mime_type for structured JSON output.
- Use count_tokens before sending a request to estimate input token usage.
Intro to Request and Response Logging with Gemini
- Use sampling_rate to control the fraction of requests logged.
- Use an automatically created BigQuery destination by passing bq://PROJECT_ID.
- Handle AlreadyExists when enabling logging.
- Restore the original sampling_rate after temporarily updating the config to retrieve output_uri.
- Quote the full BigQuery table ID before querying.
Introduction to Long Context Window with Gemini on Vertex AI
- Start by putting all relevant tokens into the context window when the input fits, while recognizing RAG and summarization can still be relevant.
- Use count_tokens() before generate_content() for long-context requests.
- Use Part.from_uri with explicit MIME types for text, video, and audio files.
- Inspect response.usage_metadata after generation.
- Use context caching to reduce time and cost for repeated large-context requests.
Getting Started with Model Optimizer
- Use environment variables for project and region fallbacks.
- Set routing preferences explicitly with ModelSelectionConfig.
- Keep system instruction and model selection config inside GenerateContentConfig.
- Use streaming when incremental response display is desired.
- Declare function parameters and response schemas before passing tools to the model.
Getting Started with the Live API Native Audio
- Use a reusable configuration helper for transcription, proactivity, affective dialog, and system instructions.
- Use an async context manager for the Live API session lifecycle.
- Send conversation turns sequentially to maintain session context.
- Collect input and output transcriptions alongside streamed audio when transcription is enabled.
- Use system instructions to constrain proactive chime-in behavior and set response persona.
Getting Started with Gemini Live API using WebSocket
- Set PROJECT_ID and LOCATION before constructing the model path.
- Use the regional aiplatform.googleapis.com WebSocket endpoint for non-global locations.
- Send setup once, await the setup response, then start streaming.
- Run send_loop and receive_loop concurrently with asyncio.gather for bidirectional interaction.
- Use realtime_input for high-frequency audio and video chunks and client_content for discrete text turns.
- Use the Google Gen AI SDK for a simplified Live API session and interruption handling.
- Use WebSockets when you need direct control over the handshake and raw JSON payloads.
- Stream audio in small chunks and delay briefly to simulate real-time input from a microphone.
- Set the input MIME type to audio/pcm;rate=16000 when sending audio.
- Decode inline audio data and concatenate int16 PCM chunks before playback.
Getting Started with LangChain 🦜️🔗 + Gemini API in Vertex AI
- Use PromptTemplate for prompts with runtime input variables.
- Use StructuredOutputParser format instructions when structured model output is needed.
- Split long documents with RecursiveCharacterTextSplitter before embedding or summarization.
- Use retrievers over vector stores to combine documents with language models.
- Use ConversationBufferMemory when a conversation chain needs prior message context.
Prompt Design - Best Practices
- Be concise in prompts.
- Be specific and well-defined.
- Ask one task at a time.
- Use system instructions to guardrail the model from irrelevant responses.
- Turn generative tasks into classification tasks to reduce output variability.
Get Started with Vertex AI Prompt Optimizer
- Use zero-shot optimization for rapid prompt refinement when no evaluation dataset is available.
- Use data-driven optimization when sample inputs and expected outputs define what better performance means.
- Provide examples where the current system instruction performs poorly for prompt optimization.
- Use 50-100 distinct samples for reliable prompt optimization results.
- Include a target field for computation-based metrics such as question_answering_correctness.
Intro to Building a Scalable and Modular RAG System with RAG Engine in Vertex AI
- Use an explicit embedding model when creating the RAG corpus.
- Configure chunk_size and chunk_overlap during bulk imports.
- Use top_k and vector_distance_threshold to tune retrieval scope.
- Run rag.retrieval_query to inspect retrieved context directly.
- Pass a RAG retrieval tool into generate_content to ground model responses.
Responsible AI with Gemini API in Vertex AI: Safety ratings and thresholds
- Inspect safety_ratings instead of relying only on generated text.
- Test prompts against safety categories before deployment.
- Set safety thresholds according to business policies and use case needs.
- Use low-variability generation settings when comparing safety behavior.
- Check finish_reason to understand why generation stopped.
Intro to thought signatures with REST API
- Set GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION before constructing the endpoint.
- Use function_declarations with explicit parameters and required fields.
- Include thought signatures when sending function execution results back to the model.
- Keep the full conversation history when requesting the final response.
- Clean up generated response*.json files after the tutorial.
- Install or upgrade google-genai before running the notebook.
- Authenticate Colab users with auth.authenticate_user().
- Use a Google Cloud Project for authentication in the tutorial workflow.
- Inspect url_context_metadata to debug retrieved sources and verify information sources.
- Inspect grounding_metadata when using Google Search grounding.
Import, Deploy, and Serve custom open models on Vertex AI using Vertex AI Model Garden SDK.
- Use environment variables as fallbacks for project and region.
- Enable hf_transfer for faster Hugging Face downloads.
- Upload large model files to Cloud Storage with parallel chunk transfers.
- Call list_deploy_options before deployment to verify supported configurations and resource needs.
- Use a dedicated endpoint for prediction calls.
Getting started with Google Generative AI using the Gen AI SDK
- Read project and region from GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION when notebook parameters are unset.
- Use GenerateContentConfig for system instructions, parameters, safety settings, tools, schemas, and cached content.
- Use Part.from_uri for model-readable Cloud Storage or HTTPS file inputs.
- Inspect response.candidates[0].safety_ratings after applying safety filters.
- Use response_mime_type application/json with response_schema for controlled generation.
Configuring Retries in the Gen AI SDK
- Configure retries for transient network errors, temporary server unavailability, and rate limits.
- Use exponential backoff settings such as initial_delay, attempts, exp_base, max_delay, and jitter for fine-grained retry control.
- Retry only selected HTTP status codes such as 429, 500, 502, 503, and 504.
- Apply retries at the client level when all calls should share one policy.
- Apply retries at the request level when only one call needs custom behavior.
- Fallback to the GOOGLE_CLOUD_PROJECT environment variable when PROJECT_ID is not supplied.
- Use ClientOptions to select a regional Discovery Engine endpoint for non-global locations.
- Include citations in both search summaries and generated answers.
- Use query expansion and spell correction in the search request.
- Use query rephrasing and query classification in the answer request.
Ingestion of Unstructured Documents with Metadata in Vertex AI Search
- Use layout-based chunking with a 500 token chunk size and ancestor headings when accuracy on complex documents matters.
- Provide an explicit metadata schema when fields must be retrievable, indexable, searchable, filterable, or specially mapped.
- Represent each document as JSONL with id, structData, content.mimeType, and content.uri.
- Poll long-running operations for schema updates and document imports before relying on the datastore.
- Keep documents, metadata, and generated JSONL in the same temporary bucket when cleanup simplicity matters.
Getting Started with Translation
- Use environment variables for project ID and region when notebook parameters are not provided.
- Set source_language_code and target_language_code explicitly.
- Use client.common_location_path for regional Translation API resource paths.
- Use glossaries to consistently translate domain-specific words and phrases.
- Check supported language codes and supported content formats in the Translation API documentation.
- Set safety_filter_level and person_generation in generation and editing configs.
- Use RawReferenceImage for the source image and MaskReferenceImage for mask-based edits.
- Use MASK_MODE_USER_PROVIDED when supplying your own mask image.
- Use an empty prompt for inpainting removal requests where the object should simply be removed.
- Display original and edited images side by side for visual comparison.
- Install or upgrade google-genai before using the notebook.
- Authenticate only in Colab by checking for google.colab in sys.modules.
- Keep generation and upscaling model IDs in variables.
- Use types.GenerateImagesConfig for aspect ratio, image count, and image size.
- Use types.Image.from_file for local files and types.Image(gcs_uri=…) for Cloud Storage images.
- Use asset reference images for subjects, objects, or scenes that should appear in the final video.
- Set aspect_ratio, number_of_videos, duration_seconds, resolution, person_generation, and generate_audio explicitly.
- Poll the operation before reading generated_videos.
- Use Cloud Storage URIs for multiple reference images when avoiding local downloads.
- All Veo videos include SynthID digital watermarking.
Virtual Try-On: Image Generation
- Use Image.from_file for local person and product images.
- Save an intermediate try-on output locally when it will be reused in a later request.
- Set output_mime_type and number_of_images explicitly in RecontextImageConfig or GenerateImagesConfig.
- Use safety_filter_level explicitly when generating or recontextualizing images.
- Combine multiple clothing items into one photo when trying on multiple items in a single request.
Back to Getting Started · Best Practices Map