Evaluation

Evaluation on Google Cloud is the practice of measuring generative AI model, prompt, RAG, multimodal, structured-output, and agent behavior using Vertex AI Gen AI Evaluation, custom metrics, autoraters, experiments, and scalable batch workflows. It combines model-based, computation-based, rubric-based, pairwise, trajectory, and task-specific metrics to compare systems and inspect failures before production use.

Synthesized across 77 notebooks. See the source pages under Notebooks below for line-level grounding.

Google Cloud evaluation workflows center on Vertex AI Gen AI Evaluation SDK and Service, where engineers prepare datasets, run or import model responses, define metrics, and inspect both aggregate scores and row-level explanations. The notebooks repeatedly emphasize controlled comparisons: use the same dataset, prompt template, metrics, project, and region when comparing models, prompts, RAG answer sets, or third-party systems.

Evaluation spans more than text quality. The material covers RAG groundedness and retrieval quality, structured JSON extraction, translation metrics, image and video Gecko-style rubric validation, tool-use evaluation, multi-turn agent simulation, agent trajectory metrics, and anomaly detection with traces, embeddings, BigQuery, and Vector Search.

The recurring engineering pattern is to start small, make outputs and schemas explicit, collect detailed traces or row-level results, then scale with Cloud Storage, BigQuery, Batch Prediction, Dataflow, or persisted evaluation runs. Demos often use small datasets, but several notebooks recommend around 100 examples for more reliable aggregate metrics.

Canonical workflow

1. Enable the Vertex AI API, authenticate, configure PROJECT_ID, LOCATION, and any required Cloud Storage or BigQuery destinations.
2. Create an evaluation dataset with the required columns for the task, such as prompt, response, reference, context, tool calls, predicted_trajectory, reference_trajectory, or multimodal file_data URIs.
3. Generate responses with a target model or agent, or bring precomputed responses when using BYOD evaluation, batch evaluation, or unsupported direct multimodal inference paths.
4. Define the evaluation metrics: predefined pointwise, pairwise, rubric, translation, groundedness, safety, trajectory, tool-use, computation-based, or CustomMetric metrics with explicit criteria and rubrics.
5. Run EvalTask, Gen AI Evaluation Service, batch_evaluate, Batch Prediction, Agent Evaluation, or a pipeline workflow depending on scale and artifact requirements.
6. Inspect summary metrics, row-level metrics, explanations, raw autorater output, parsed custom outputs, generated rubrics, and failed or mismatched cases.
7. Compare runs under consistent experiments, prompt templates, datasets, and metrics; tune prompts, models, thresholds, rubrics, or agent logic based on evidence.
8. Persist or retrieve results when needed, poll long-running jobs to terminal state, and clean up experiments, remote agents, buckets, pipeline jobs, and TensorBoard-backed runs when finished.

Best practices

See Evaluation - Best Practices for the full distilled guide.

• Use a fixed dataset, metric set, prompt template, and experiment structure when comparing models, prompts, RAG outputs, or third-party systems.
• Initialize Vertex AI with explicit project and location values, preferably with GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION fallbacks.
• Keep evaluation datasets structured and task-specific, with columns such as prompt, response, reference, context, instruction, intermediate_events, predicted_trajectory, and reference_trajectory as required.
• Run inference before evaluation when agent or model evaluation needs generated response columns, intermediate events, traces, or tool-call outputs.
• Define clear metric criteria and rating rubrics before running model-based or custom evaluations, and inspect MetricPromptTemplateExamples before customizing templates.
• Request structured JSON from autoraters and classifiers with response_mime_type, response_schema, or strict output formats, then parse and validate results explicitly.

Key models & APIs

• Vertex AI Gen AI Evaluation Service, Vertex AI Gen AI Evaluation SDK, Vertex AI EvalTask, Vertex AI Agent Engine, Vertex AI Agent Evaluation, Vertex AI Batch Prediction API, Vertex AI Pipelines, Dataflow ML, BigQuery, Cloud Storage JSONL evaluation inputs and outputs, Vector Search, Google Gen AI SDK, google-cloud-aiplatform[evaluation], CustomMetric, PointwiseMetric, PairwiseMetric, CustomOutputConfig, MetricPromptTemplateExamples, FunctionDeclaration and tool-call evaluation metrics, Gecko-style image and video rubric evaluation, gemini-2.5-flash, gemini-2.5-pro, gemini-2.0-flash, gemini-2.0-flash-lite, gemini-embedding-001, text-embedding-004, text-bison@001, gemini-3.5-flash, gemini-3.1-pro-preview, meta/llama-3.1-8b-instruct-maas, meta/llama-3.1-70b-instruct-maas, claude-3-5-sonnet@20240620, gpt-5-mini, gpt-4o, deepseek-ai/deepseek-r1-0528-maas

Notebooks (77)

Embeddings & Vector Search (2)

• 🛡️ AI Brand Safety: Three-Tier Agent Anomaly Detection — Builds ADK agent anomaly detection with Gemini baselines, Vector Search scoring, and tiered audits.
• Using “task type” embeddings for improving RAG search quality — Shows how task-type embeddings improve RAG Q&A retrieval quality and MRR.

Gemini (65)

• Evaluate a CrewAI agent on Vertex AI Agent Engine (Customized template) — Evaluates a CrewAI Gemini agent on Vertex AI Agent Engine with tool, trajectory, and response metrics.
• Evaluating a LangChain Agent on Vertex AI Agent Engine (Prebuilt template) — Deploys and evaluates a LangChain Gemini agent on Vertex AI Agent Engine using Gen AI Evaluation.
• Evaluate a LangGraph agent on Vertex AI Agent Engine (Customized template) — Builds a Gemini LangGraph agent on Agent Engine and evaluates tools, trajectories, and responses.
• Guardrail Classifier Agent — Builds a LangGraph guardrail classifier agent for Cymbal retail chat using Gemini on Vertex AI.
• Create & Deploy Agent and Run Gen AI Agent Evaluation — Creates, deploys, runs, and evaluates an ecommerce ADK agent on Vertex AI Agent Engine.
• Create a Gen AI Agent Evaluation for a Deployed Agent — Runs inference and creates a persisted Gen AI Agent Evaluation for a deployed Vertex AI agent.
• Bring-Your-Own-Autorater using CustomMetric — Evaluates Gemini prompts with a client-side CustomMetric and a BYO Gemini autorater.
• Bring your own computation-based CustomMetric — Evaluates Gemini outputs with locally defined computation-based Vertex AI CustomMetric functions.
• Migrate from PaLM to Gemini model — Compares PaLM text-bison and gemini-2.5-flash with Vertex AI EvalTask for summarization migration.
• Compare Generative AI Models — Evaluates Gemini models on summarization with Vertex AI Gen AI Evaluation Service.
• Customize Model-based Metrics to Evaluate a Gen AI model — Shows how to customize Vertex AI Gen AI Evaluation model-based metrics for Gemini summarization outputs.
• Enhancing quality and explainability with Vertex AI Evaluation — Ranks Gemini answers with Vertex AI pairwise and pointwise evaluation explanations.
• Evaluate agent final answer with custom parsing — Evaluates agent final answers using Vertex AI EvalTask with a custom JSON parser for autorater output.
• Evaluate your autorater with meta-evaluation — Meta-evaluates Gemini autoraters on RewardBench with agreement and correlation metrics.
• Evaluate Generative Model Tool Use — Evaluates Gemini function calling and saved tool-call predictions with Vertex AI EvalTask metrics.
• Evaluate groundedness with custom parsing — Evaluates Gemini response groundedness with Vertex AI EvalTask and custom JSON parsing.
• Evaluate images with Gecko — Evaluates prompt-image alignment with Gecko-style rubric generation and VQA validation in Vertex AI.
• Evaluate LangChain — Evaluates a LangChain recipe chatbot with Vertex AI Rapid Evaluation and custom Gemini-based metrics.
• Evaluating multimodal task — Evaluates image-grounded car damage labels with Vertex AI EvalTask and a Gemini custom autorater.
• Evaluate generated answers from Retrieval-Augmented Generation (RAG) using Rapid Evaluation and Dataflow ML with Vertex AI pipelines — Builds a Vertex AI Pipeline to batch-evaluate RAG Q&A outputs with Rapid Eval API and Dataflow ML.
• Evaluate Generated Answers from Retrieval-Augmented Generation (RAG) for Question Answering with Gen AI Evaluation Service SDK — Evaluates BYO RAG QA answers with Vertex AI Gen AI Evaluation, custom metrics, and result visualizations.
• Evaluate a Translation Model — Evaluates stored translation responses with Vertex AI EvalTask using BLEU, COMET, and MetricX.
• Evaluate videos with Gecko — Uses Vertex AI evaluation to run Gecko-style rubric generation and video validation.
• Evaluating Agents - Evaluate a CrewAI agent with Vertex AI Gen AI Evaluation Service — Evaluates a CrewAI product agent with Vertex AI Gen AI Evaluation metrics and BYOD evaluation data.
• Evaluating Agents - Evaluate a LangGraph agent with Vertex AI Gen AI Evaluation Service — Evaluates a LangGraph Gemini agent with Vertex AI Gen AI Evaluation metrics and BYOD data.
• Evaluating prompts at scale with Gemini Batch Prediction API — Evaluates Gemini image-classification prompts at scale with Batch Prediction and BigQuery.
• Intro to Batch Evaluations with the Gemini API — Runs asynchronous batch evaluation of Gemini responses with Vertex AI and Cloud Storage.
• Getting Started with Vertex AI Python SDK for Gen AI Evaluation Service — Defines a custom pointwise metric and evaluates stored LLM responses with Vertex AI Gen AI Evaluation Service.
• Gen AI Evaluation Service SDK Preview-to-GA Migration Guide — Migrates Vertex AI Gen AI Evaluation SDK preview patterns to GA EvalTask, pointwise, and pairwise metrics.
• Rubric evaluation - Multimodal and Custom metric for text quality — Evaluates multimodal car-damage responses and text summaries with Vertex AI rubric-based metrics
• Evaluate and Optimize Prompt Template Design for Better Results — Compares Gemini prompt templates using Vertex AI EvalTask metrics to choose the best summarization prompt.
• Rubric-based instruction following evaluation using Gen AI Evaluation Service — Evaluates Gemini instruction following with rubric-based metrics in Vertex AI.
• Evaluate Gemini Structured Output — Evaluates Gemini structured JSON extraction from scanned order forms with Vertex AI Gen AI Evaluation.
• Evaluate images with predefined Gecko — Evaluates text-to-image outputs with Vertex AI predefined Gecko image rubrics.
• Use Gen AI Evaluation SDK to Evaluate Models in Vertex AI Studio, Model Garden, and Model Registry — Evaluates Gemini, Llama MaaS, Claude, and prompt templates with Vertex AI Gen AI Evaluation SDK.
• Evaluate videos with predefined Gecko — Evaluates generated videos with Vertex AI Gecko text-to-video rubrics.
• Evaluate Generative Model Tool Use with Custom Code Execution — Evaluates Gemini tool-use outputs with Vertex AI remote custom metrics.
• Evaluate your ADK agent using Vertex AI Gen AI Evaluation service — Evaluates an ADK product research agent with Vertex AI Gen AI Evaluation metrics.
• Using Gen AI Evaluation SDK for Google Observability Gen AI multi-modal datasets — Evaluates Google Observability multimodal Gen AI data from GCS using Vertex AI Gen AI Evaluation.
• Evaluating Third-Party LLMs with the Vertex AI Gen AI Evaluation SDK — Evaluates third-party, MaaS, BYOM, and Gemini models with Vertex AI Gen AI Evaluation.
• Migrating Foundation Models: A Practical Guide with Gen AI Evaluation Serivce — Compares model migration candidates with Vertex AI Gen AI Evaluation and prompt optimization workflows.
• Overview — Evaluates a CrewAI research crew with Phoenix tracing and Vertex AI trajectory metrics.
• Gen AI Eval - Multi-turn Agent Eval, User Simulation, Metric Registration, Auto-Loss Analysis — Evaluates ADK travel agents with Vertex Gen AI Eval multi-turn simulation, custom metrics, and loss analysis.
• Getting Started: Quick Gen AI Evaluation — Evaluates gemini-2.5-flash responses with Vertex AI Gen AI Eval Service using a default quality rubric.
• View Gen AI Agent Evaluation Run Results — Retrieves a Vertex AI Gen AI evaluation run and displays its embedded report.
• Intro to Logprobs — Introduces Gemini logprobs on Vertex AI for confidence, autocomplete, and RAG grounding analysis.
• Question Answering with Generative Models on Vertex AI — Shows prompt patterns for Gemini question answering on Vertex AI, including simple fuzzy evaluation.
• Text Classification with Generative Models on Vertex AI — Classifies text with Gemini on Vertex AI using zero-shot, few-shot, and evaluation workflows.
• Text Summarization with Generative Models on Vertex AI — Demonstrates Gemini text summarization prompts on Vertex AI and evaluates summaries with ROUGE.
• Get Started with Vertex AI Prompt Optimizer — Shows zero-shot and data-driven prompt optimization with Vertex AI Prompt Optimizer.
• Get started with Vertex Prompt Optimizer - Custom metric — Optimizes a Gemini prompt with Vertex AI Prompt Optimizer using a custom Cloud Function metric.
• Get Started with Vertex AI Prompt Optimizer - Long prompt — Runs Vertex AI Prompt Optimizer to improve a long Gemini prompt using data-driven evaluation.
• Get Started with Vertex AI Prompt Optimizer - Multimodality — Optimizes a multimodal Gemini prompt with Vertex AI Prompt Optimizer on MathVista image QA.
• Get Started with Vertex AI Prompt Optimizer - Tool usage — Optimizes a Gemini tool-calling system instruction with Vertex AI Prompt Optimizer data-driven mode.
• Vertex Prompt Optimizer Notebook UI (Preview) — Shows how to configure, run, and inspect Vertex AI Prompt Optimizer jobs from a notebook UI.
• Evaluating Vertex RAG Engine Generation with Vertex AI Python SDK for Gen AI Evaluation Service — Evaluates Vertex AI RAG Engine responses with a custom Gen AI Evaluation Service metric.
• Advanced RAG Techniques - Vertex RAG Engine Retrieval Quality Evaluation and Hyperparameters Tuning — Evaluates Vertex AI RAG Engine retrieval quality and tunes chunking, top-k, threshold, and embedding settings.
• Prepare High-Quality Preference Data for Gemini 2.5 — Prepares and filters Gemini preference data with Vertex AI Gen AI Evaluation SDK for DPO and SFT.
• Supervised Fine-Tuning with integrated Gen AI Evaluation — Fine-tunes gemini-2.5-flash with automatic Gen AI Evaluation metrics at each checkpoint.
• Integrate Custom Metrics into Gemini Supervised Fine-Tuning — Adds a Python custom metric to Gemini SFT using Vertex AI REST tuning jobs and GCS outputs.
• Supervised Fine Tuning with Gemini 2.5 Flash for Image Captioning — Fine-tunes Gemini 2.5 Flash on GCS image-caption pairs and evaluates ROUGE before and after tuning.
• Supervised fine-tuning with Gemini 2.0 Flash for Q&A using the Google Gen AI SDK — Fine-tunes Gemini 2.0 Flash on SQuAD Q&A with Vertex AI supervised tuning and evaluates EM/F1 gains.
• Supervised Fine Tuning with Gemini 2.5 Flash for Article Summarization — Fine-tunes Gemini 2.5 Flash on WikiLingua article summaries and evaluates ROUGE before and after tuning.
• Evaluate — Evaluates Gemini-based document classification on image samples and visualizes confusion matrices.
• Know Your Customer Use Case - Gemini Grounding with Google Search — Builds grounded Gemini KYC negative-news reports and evaluates response quality with custom Vertex AI metrics.

Open Models (6)

• Accelerate LLM Inference with EAGLE Speculative Decoding on Vertex AI — Benchmarks EAGLE speculative decoding for Llama 4 Scout on Vertex AI against a baseline endpoint.
• Qwen 3 evaluation - Bring your own data eval — Compares fine-tuned and base Qwen 3 medical summaries using Vertex AI pairwise evaluation.
• Using open autorater for running evaluations with Vertex AI Gen AI Evaluation — Deploys Selene as an open judge on Vertex AI and uses Gen AI Evaluation to score LLM responses.
• Hugging Face DLCs: Using Gemma for running evaluations with Vertex AI Gen AI Evaluation — Deploys Gemma 2 on Vertex AI TGI and evaluates summarization with Gen AI Evaluation.
• MetaMath with Vertex AI Open Source Model Tuning — Fine-tunes a Llama 3.1 8B model on MetaMathQA using Vertex AI managed tuning.
• Build and deploy a Hugging Face smolagent using DeepSeek-r1 on Vertex AI — Deploys DeepSeek R1 Distill Qwen 7B on Vertex AI and wraps it in a smolagents math verifier agent.

Vertex AI Search (3)

• Clearbox for Ranking Tuning — Tunes Vertex AI Search ranking with ClearBox using BEIR FIQA signals and recall-based validation.
• Gemini Enterprise answer eval using BLEU, ROUGE, BERT, Similarity Score — Evaluates Gemini Enterprise answers against a golden dataset with NLP metrics and saves results.
• Setup — Evaluates Discovery Engine semantic ranking on BEIR datasets with NDCG and ROC AUC metrics.

Tools (1)

• Tutorial for Running Prompt Management and Evaluation — Runs prompt management, MathVista evaluation, and Vertex AI prompt optimization in a Streamlit app.

Related concepts

• Prompt Engineering · Agents & ADK · Tuning & Customization · RAG & Grounding · Applied Use Cases · Vision · Open & Partner Models · Function Calling & Tools

---

Part of the Generative AI Google Brain · Evaluation - Best Practices