Prompt Engineering

Prompt Engineering on Google Cloud is the practice of designing, testing, evaluating, and optimizing prompts for Gemini and related Vertex AI generative models so they produce reliable text, structured, multimodal, audio, video, tool-calling, and agent outputs. In these notebooks, it spans prompt design patterns, controlled generation, evaluation, optimization, grounding, orchestration, safety mitigations, and task-specific workflows on Vertex AI, Agent Platform, and the Gemini API.

Synthesized across 41 notebooks. See the source pages under Notebooks below for line-level grounding.

Prompt engineering starts with clear task framing: specify the goal, required format, allowed context, constraints, examples, and generation parameters. The notebooks repeatedly emphasize concise, specific prompts, one task at a time, low temperature for deterministic extraction/classification/security-sensitive workflows, and explicit grounding when answers must come only from provided context.

For application use, prompts are not treated as static text. They are versioned, templated, evaluated against datasets, compared across prompt variants, constrained with JSON schemas or enums, and optimized with Vertex AI Prompt Optimizer when examples and metrics are available.

The area also extends beyond text. The same discipline applies to multimodal image, video, speech, RAG, tool-use, SQL agents, code refactoring, security guardrails, and evaluation pipelines, with Cloud Storage, BigQuery, LangChain, LlamaIndex, Vertex AI Evaluation, Prompt Management, and Batch Prediction used to operationalize prompt iteration at scale.

Canonical workflow

1. Define the task, target user outcome, input sources, output format, safety requirements, and whether the answer must be grounded only in provided context.
2. Choose the appropriate Gemini or Vertex AI model and initialize the Google Cloud project, location, authentication, and required APIs.
3. Write a concise prompt or system instruction with explicit constraints, examples when useful, and generation parameters suited to the task.
4. Constrain outputs for application use with response_mime_type, response_schema, response_json_schema, enums, closed vocabularies, or parser format instructions.
5. Test on a small representative input before scaling to batch prediction, RAG, agents, multimodal files, or optimization jobs.
6. Evaluate against ground truth, rubrics, custom metrics, human ratings, or built-in Vertex AI Gen AI Evaluation metrics using consistent datasets and model settings.
7. Compare prompt variants, inspect explanations and raw outputs, then optimize with Vertex AI Prompt Optimizer or manual prompt iteration.
8. Promote the selected prompt through prompt management, store traceability metadata, monitor cost and quotas, and clean up billable resources created during experimentation.

Best practices

See Prompt Engineering - Best Practices for the full distilled guide.

• Be concise, specific, grammatically clear, and ask one task at a time.
• Use system instructions to set durable behavior, guardrails, persona boundaries, output restrictions, and source-of-truth rules.
• Use zero-shot, one-shot, or few-shot prompting intentionally; keep examples representative and avoid overloading the prompt with too many examples.
• For grounded Q&A, extraction, RAG, and agents, explicitly instruct the model to use provided context, retrieved documents, tools, or visible evidence instead of prior knowledge.
• Use low temperature or temperature 0 for deterministic classification, extraction, SQL generation, medical coding, security demonstrations, and repeatable evaluations.
• For structured outputs, set response_mime_type to application/json and use response_schema, response_json_schema, Pydantic schemas, enums, or closed vocabularies where appropriate.

Key models & APIs

• gemini-2.5-flash, gemini-2.0-flash, gemini-3.5-flash, gemini-3.1-pro-preview, gemini-3.1-flash-lite, gemini-3.1-flash-tts-preview, gemini-embedding-001, text-embedding-005, google/text-embedding-005, Vertex AI Gemini API and google-genai SDK, Vertex AI Gen AI Evaluation SDK, EvalTask, CustomMetric, and rubric metrics, Vertex AI Prompt Optimizer, Gemini Batch Prediction API, Vertex AI Prompt Management, LangChain and LlamaIndex integrations, Vertex AI Vector Search, BigQuery and BigQuery DataFrames ML, Cloud Storage, Cloud DLP and Cloud Natural Language API, Veo 3.1 video generation models, Vertex AI MaaS reasoning models through the OpenAI SDK

Notebooks (41)

Audio (1)

• Gemini 3.1 Flash Text-to-Speech Generation — Shows Gemini 3.1 Flash TTS generation with voices, languages, speakers, and audio tags.

Gemini (34)

• Intro to Structured Output with the Gemini API — Shows how to make Gemini 2.5 Flash return JSON, enums, and schema-constrained multimodal outputs.
• Bring-Your-Own-Autorater using CustomMetric — Evaluates Gemini prompts with a client-side CustomMetric and a BYO Gemini autorater.
• Evaluate LangChain — Evaluates a LangChain recipe chatbot with Vertex AI Rapid Evaluation and custom Gemini-based metrics.
• Evaluating prompts at scale with Gemini Batch Prediction API — Evaluates Gemini image-classification prompts at scale with Batch Prediction and BigQuery.
• Rubric evaluation - Multimodal and Custom metric for text quality — Evaluates multimodal car-damage responses and text summaries with Vertex AI rubric-based metrics
• Evaluate and Optimize Prompt Template Design for Better Results — Compares Gemini prompt templates using Vertex AI EvalTask metrics to choose the best summarization prompt.
• Use Gen AI Evaluation SDK to Evaluate Models in Vertex AI Studio, Model Garden, and Model Registry — Evaluates Gemini, Llama MaaS, Claude, and prompt templates with Vertex AI Gen AI Evaluation SDK.
• Getting Started with LangChain 🦜️🔗 + Gemini API in Vertex AI — Introduces LangChain with Gemini on Vertex AI for prompts, embeddings, retrieval, memory, and chains.
• Chain of Thought & ReAct — Demonstrates CoT prompting and ReAct agents with Vertex AI, LangChain, Wikipedia, and BigQuery.
• Question Answering with Generative Models on Vertex AI — Shows prompt patterns for Gemini question answering on Vertex AI, including simple fuzzy evaluation.
• Text Classification with Generative Models on Vertex AI — Classifies text with Gemini on Vertex AI using zero-shot, few-shot, and evaluation workflows.
• Text Extraction with Generative Models on Vertex AI — Uses Gemini on Vertex AI to extract structured facts from text with constrained and few-shot prompts.
• Text Summarization with Generative Models on Vertex AI — Demonstrates Gemini text summarization prompts on Vertex AI and evaluates summaries with ROUGE.
• Prompt Design - Best Practices — Demonstrates prompt design best practices for Gemini on Vertex AI with google-genai.
• Get Started with Vertex AI Prompt Optimizer — Shows zero-shot and data-driven prompt optimization with Vertex AI Prompt Optimizer.
• Get started with Vertex Prompt Optimizer - Custom metric — Optimizes a Gemini prompt with Vertex AI Prompt Optimizer using a custom Cloud Function metric.
• Get Started with Vertex AI Prompt Optimizer - Long prompt — Runs Vertex AI Prompt Optimizer to improve a long Gemini prompt using data-driven evaluation.
• Get Started with Vertex AI Prompt Optimizer - Multimodality — Optimizes a multimodal Gemini prompt with Vertex AI Prompt Optimizer on MathVista image QA.
• Get Started with Vertex AI Prompt Optimizer - Tool usage — Optimizes a Gemini tool-calling system instruction with Vertex AI Prompt Optimizer data-driven mode.
• Vertex Prompt Optimizer Notebook UI (Preview) — Shows how to configure, run, and inspect Vertex AI Prompt Optimizer jobs from a notebook UI.
• Gen AI & LLM Security for developers — Shows prompt injection attacks against Gemini and layered mitigations with DLP, NL API, safety filters, and embeddings.
• Supervised Fine Tuning with Gemini 2.5 Flash for Article Summarization — Fine-tunes Gemini 2.5 Flash on WikiLingua article summaries and evaluates ROUGE before and after tuning.
• BigQuery DataFrames ML: Prescription Drug Name Generation — Generates pharmaceutical brand name ideas with BigQuery DataFrames ML and Gemini.
• Code Vulnerability Scanning & Automated Remediation using Gemini API in Vertex AI (Gemini 2.0) — Scans Python files from GCS with Gemini 2.0 Flash and exports vulnerability reports to CSV and JSON.
• Reduce Tech Debt with Gemini 3 Pro — Uses Gemini 3.1 Pro on Vertex AI to refactor legacy Python code through tests, specs, design docs, and verification.
• Text Summarization of Large Documents using LangChain 🦜🔗 — Summarizes large PDFs with LangChain and Gemini using stuff, map-reduce, and refine chains.
• ReAct (Reasoning + Acting) + Custom tool for Healthcare NL API + Gemini 2.0 + LangChain — Builds a LangChain ReAct agent using Gemini 2.0 and Healthcare NLP to suggest medical billing codes.
• Building Knowledge Graphs with Gemini — Extracts knowledge graphs from text and PDF documents with Gemini using deterministic prompts and TSV outputs.
• Video Captioning with Gemini — Uses Gemini on Vertex AI to generate rich multimodal captions from a video in Cloud Storage.
• Slide Generation with Gemini and Marp — Uses Gemini and Marp to turn a blog post into a Markdown slide deck and PDF.
• Multimodal retail recommendation: using Gemini to recommend items based on images and image reasoning — Builds a Gemini multimodal retail recommender that explains chair fit for a room image.
• Product attributes extraction and detailed descriptions from images using Gemini 2.0 — Extract product attributes and detailed retail descriptions from images with Gemini 2.0 on Vertex AI.
• Leverage LlamaIndex with Vertex AI Vector Search to perform question answering RAG — Builds LlamaIndex RAG on Vertex AI Vector Search, compares prompts, and adds multi-document agents.
• Unlocking Multimodal Video Transcription with Gemini — Uses Gemini to build multimodal video transcripts with voice IDs, speaker names, and structured JSON output.

Open Models (2)

• Handling Reasoning with MaaS Models on Vertex AI using vLLM — Shows how to call reasoning MaaS models on Vertex AI through the OpenAI SDK.
• Guess who or what app using Hugging Face Deep Learning container model on Vertex AI — Builds a Gradio riddle game using Gemini and a Hugging Face FLUX model deployed on Vertex AI.

Vertex AI Search (1)

• Gemini Enterprise custom agent with prompt management — Builds a Gemini Enterprise ADK SQL agent using Vertex AI Prompt Management and schema file context.

Tools (1)

• Tutorial for Running Prompt Management and Evaluation — Runs prompt management, MathVista evaluation, and Vertex AI prompt optimization in a Streamlit app.

Vision (1)

• Veo 3.1 Video Generation — Generates Veo 3.1 videos from text and images with optional Gemini prompt optimization.

Workshops (1)

• AI Agents for Engineers (Evolution of AI Agents) — Compares zero-shot, LangChain, and LangGraph essay generation with Gemini and Tavily search.

Related concepts

• Evaluation · Gemini Capabilities · Applied Use Cases · Vision · Getting Started · Function Calling & Tools · Open & Partner Models · Tuning & Customization

---

Part of the Generative AI Google Brain · Prompt Engineering - Best Practices