Gen AI Evaluation Service

Vertex AI's framework for evaluating generative models and agents with model-based and computation-based metrics, including pointwise and pairwise judging. · Official docs

Related concepts

• Getting Started · Gemini Capabilities · Prompt Engineering · Function Calling & Tools · Agents & ADK · Agent Engine · RAG & Grounding · Vision · Image & Video Generation · Evaluation · Tuning & Customization · Open & Partner Models · Translation · MLOps & Deployment · Applied Use Cases

Used in 43 notebooks

Gemini (39)

• Evaluating a LangChain Agent on Vertex AI Agent Engine (Prebuilt template) — Deploys and evaluates a LangChain Gemini agent on Vertex AI Agent Engine using Gen AI Evaluation.
• Evaluate a LangGraph agent on Vertex AI Agent Engine (Customized template) — Builds a Gemini LangGraph agent on Agent Engine and evaluates tools, trajectories, and responses.
• Create & Deploy Agent and Run Gen AI Agent Evaluation — Creates, deploys, runs, and evaluates an ecommerce ADK agent on Vertex AI Agent Engine.
• Bring-Your-Own-Autorater using CustomMetric — Evaluates Gemini prompts with a client-side CustomMetric and a BYO Gemini autorater.
• Bring your own computation-based CustomMetric — Evaluates Gemini outputs with locally defined computation-based Vertex AI CustomMetric functions.
• Migrate from PaLM to Gemini model — Compares PaLM text-bison and gemini-2.5-flash with Vertex AI EvalTask for summarization migration.
• Compare Generative AI Models — Evaluates Gemini models on summarization with Vertex AI Gen AI Evaluation Service.
• Customize Model-based Metrics to Evaluate a Gen AI model — Shows how to customize Vertex AI Gen AI Evaluation model-based metrics for Gemini summarization outputs.
• Enhancing quality and explainability with Vertex AI Evaluation — Ranks Gemini answers with Vertex AI pairwise and pointwise evaluation explanations.
• Evaluate Generative Model Tool Use — Evaluates Gemini function calling and saved tool-call predictions with Vertex AI EvalTask metrics.
• Evaluate images with Gecko — Evaluates prompt-image alignment with Gecko-style rubric generation and VQA validation in Vertex AI.
• Evaluate LangChain — Evaluates a LangChain recipe chatbot with Vertex AI Rapid Evaluation and custom Gemini-based metrics.
• Evaluating multimodal task — Evaluates image-grounded car damage labels with Vertex AI EvalTask and a Gemini custom autorater.
• Evaluate Generated Answers from Retrieval-Augmented Generation (RAG) for Question Answering with Gen AI Evaluation Service SDK — Evaluates BYO RAG QA answers with Vertex AI Gen AI Evaluation, custom metrics, and result visualizations.
• Evaluate a Translation Model — Evaluates stored translation responses with Vertex AI EvalTask using BLEU, COMET, and MetricX.
• Evaluating Agents - Evaluate a CrewAI agent with Vertex AI Gen AI Evaluation Service — Evaluates a CrewAI product agent with Vertex AI Gen AI Evaluation metrics and BYOD evaluation data.
• Evaluating Agents - Evaluate a LangGraph agent with Vertex AI Gen AI Evaluation Service — Evaluates a LangGraph Gemini agent with Vertex AI Gen AI Evaluation metrics and BYOD data.
• Intro to Batch Evaluations with the Gemini API — Runs asynchronous batch evaluation of Gemini responses with Vertex AI and Cloud Storage.
• Getting Started with Vertex AI Python SDK for Gen AI Evaluation Service — Defines a custom pointwise metric and evaluates stored LLM responses with Vertex AI Gen AI Evaluation Service.
• Gen AI Evaluation Service SDK Preview-to-GA Migration Guide — Migrates Vertex AI Gen AI Evaluation SDK preview patterns to GA EvalTask, pointwise, and pairwise metrics.
• Rubric evaluation - Multimodal and Custom metric for text quality — Evaluates multimodal car-damage responses and text summaries with Vertex AI rubric-based metrics
• Evaluate and Optimize Prompt Template Design for Better Results — Compares Gemini prompt templates using Vertex AI EvalTask metrics to choose the best summarization prompt.
• Rubric-based instruction following evaluation using Gen AI Evaluation Service — Evaluates Gemini instruction following with rubric-based metrics in Vertex AI.
• Evaluate Gemini Structured Output — Evaluates Gemini structured JSON extraction from scanned order forms with Vertex AI Gen AI Evaluation.
• Use Gen AI Evaluation SDK to Evaluate Models in Vertex AI Studio, Model Garden, and Model Registry — Evaluates Gemini, Llama MaaS, Claude, and prompt templates with Vertex AI Gen AI Evaluation SDK.
• Evaluate Generative Model Tool Use with Custom Code Execution — Evaluates Gemini tool-use outputs with Vertex AI remote custom metrics.
• Evaluate your ADK agent using Vertex AI Gen AI Evaluation service — Evaluates an ADK product research agent with Vertex AI Gen AI Evaluation metrics.
• Using Gen AI Evaluation SDK for Google Observability Gen AI multi-modal datasets — Evaluates Google Observability multimodal Gen AI data from GCS using Vertex AI Gen AI Evaluation.
• Evaluating Third-Party LLMs with the Vertex AI Gen AI Evaluation SDK — Evaluates third-party, MaaS, BYOM, and Gemini models with Vertex AI Gen AI Evaluation.
• Migrating Foundation Models: A Practical Guide with Gen AI Evaluation Serivce — Compares model migration candidates with Vertex AI Gen AI Evaluation and prompt optimization workflows.
• Overview — Evaluates a CrewAI research crew with Phoenix tracing and Vertex AI trajectory metrics.
• Gen AI Eval - Multi-turn Agent Eval, User Simulation, Metric Registration, Auto-Loss Analysis — Evaluates ADK travel agents with Vertex Gen AI Eval multi-turn simulation, custom metrics, and loss analysis.
• Getting Started: Quick Gen AI Evaluation — Evaluates gemini-2.5-flash responses with Vertex AI Gen AI Eval Service using a default quality rubric.
• View Gen AI Agent Evaluation Run Results — Retrieves a Vertex AI Gen AI evaluation run and displays its embedded report.
• Evaluating Vertex RAG Engine Generation with Vertex AI Python SDK for Gen AI Evaluation Service — Evaluates Vertex AI RAG Engine responses with a custom Gen AI Evaluation Service metric.
• Prepare High-Quality Preference Data for Gemini 2.5 — Prepares and filters Gemini preference data with Vertex AI Gen AI Evaluation SDK for DPO and SFT.
• Supervised Fine-Tuning with integrated Gen AI Evaluation — Fine-tunes gemini-2.5-flash with automatic Gen AI Evaluation metrics at each checkpoint.
• Integrate Custom Metrics into Gemini Supervised Fine-Tuning — Adds a Python custom metric to Gemini SFT using Vertex AI REST tuning jobs and GCS outputs.
• Know Your Customer Use Case - Gemini Grounding with Google Search — Builds grounded Gemini KYC negative-news reports and evaluates response quality with custom Vertex AI metrics.

Open Models (4)

• Qwen 3 evaluation - Bring your own data eval — Compares fine-tuned and base Qwen 3 medical summaries using Vertex AI pairwise evaluation.
• Using open autorater for running evaluations with Vertex AI Gen AI Evaluation — Deploys Selene as an open judge on Vertex AI and uses Gen AI Evaluation to score LLM responses.
• Hugging Face DLCs: Using Gemma for running evaluations with Vertex AI Gen AI Evaluation — Deploys Gemma 2 on Vertex AI TGI and evaluates summarization with Gen AI Evaluation.
• Build and deploy a Hugging Face smolagent using DeepSeek-r1 on Vertex AI — Deploys DeepSeek R1 Distill Qwen 7B on Vertex AI and wraps it in a smolagents math verifier agent.

---

Entity · All entities · Home