MLOps & Deployment

MLOps & Deployment on Google Cloud is the practice of packaging, deploying, monitoring, evaluating, tuning, and operating generative AI models and agents across Vertex AI, Agent Platform, Cloud Run, GKE, BigQuery, and related services. It covers both managed workflows and custom serving patterns for Gemini, open models, RAG systems, batch prediction, fine-tuning, and agent runtimes.

Synthesized across 29 notebooks. See the source pages under Notebooks below for line-level grounding.

This area is about moving from notebook experimentation to repeatable cloud execution: containerized agents, managed agents, Vertex AI endpoints, Cloud Run GPU services, GKE model servers, batch pipelines, evaluation pipelines, and tuning jobs.

The recurring pattern is to make configuration explicit, store artifacts in Cloud Storage or BigQuery, validate locally before deployment, grant precise IAM roles, and poll asynchronous jobs until terminal states before consuming outputs.

Production readiness depends heavily on authentication, quota, region compatibility, service-account permissions, cleanup, and observability. Many examples are tutorials or previews, so engineers should treat them as deployment blueprints rather than turnkey production systems.

Canonical workflow

1. Set project, region, bucket, dataset, model, and service configuration explicitly through environment variables or config files.
2. Enable required APIs and verify authentication, billing, quotas, IAM roles, and service-agent permissions before creating resources.
3. Package the workload reproducibly using Dockerfiles, requirements.txt, Terraform, managed SDK configuration, or Cloud Storage model artifacts.
4. Validate locally where possible, including agent behavior, custom handlers, serving containers, tool definitions, datasets, and request schemas.
5. Deploy to the appropriate runtime: Agent Runtime, Managed Agents, Vertex AI endpoints, Vertex AI Pipelines, Cloud Run GPU, GKE, Cloud Functions, or Dataflow.
6. Monitor asynchronous operations by polling job, endpoint, pod, batch prediction, tuning, pipeline, or trace state until terminal status.
7. Inspect outputs with targeted sampling, row-level metrics, aggregate metrics, traces, logs, or prediction calls instead of assuming success.
8. Clean up endpoints, agents, buckets, datasets, repositories, clusters, Cloud Run services, pipeline jobs, and local artifacts after testing.

Best practices

See MLOps & Deployment - Best Practices for the full distilled guide.

• Keep project, region, model, bucket, endpoint, and runtime settings explicit and reusable through config files or environment variables.
• Use reproducible packaging: Dockerfile, requirements.txt, setup.py, Terraform, managed SDK deployment options, or Cloud Storage model snapshots depending on the runtime.
• Test agents, custom handlers, tool definitions, serving containers, and prediction paths locally before deploying to Agent Runtime, Agent Engine, Vertex AI, Cloud Run, or GKE.
• Store durable session state, memory, datasets, pipeline artifacts, model artifacts, and evaluation outputs outside ephemeral runtimes using managed services such as Vertex AI Sessions, Memory Bank, Cloud Storage, BigQuery, or Vertex AI Experiments.
• Use dedicated service accounts and grant explicit roles for Vertex AI, Artifact Registry, Cloud Storage, Cloud Build, Cloud Run, Cloud SQL, Secret Manager, Dataflow, GKE, and Discovery Engine as required.
• Keep production services private by default with IAM authentication; avoid tutorial-only unauthenticated Cloud Run, Toolbox, or function deployments in production.

Key models & APIs

• Vertex AI API / Agent Platform API (aiplatform.googleapis.com), Vertex AI Agent Runtime / Agent Engine, Google ADK with agent_engines.AdkApp, Managed Agents API with google-genai, google-cloud-aiplatform with agent_engines, langchain, prediction, and tuning extras, Vertex AI Sessions, VertexAISessionService, VertexAiMemoryBankService, and Memory Bank, Vertex AI Pipelines, BatchPredictionJob, VertexNotificationEmailOp, DataflowPythonJobOp, and WaitGcpResourcesOp, Rapid Evaluation API and integrated Gen AI Evaluation, BigQuery, BigFrames, Cloud Storage, Artifact Registry, Cloud Build, Secret Manager, Cloud SQL, Cloud Run, GKE, Cloud Functions, Cloud Trace, and Vertex AI Search / Discovery Engine, Vertex AI Model Garden SDK and Terraform-based Model Garden deployment, Ollama, vLLM, Hugging Face Text Generation Inference, Hugging Face PyTorch DLCs, TRL SFT, LoRA, and LiteLLM OpenAI-compatible APIs, Gemini models: gemini-3.1-flash-lite, gemini-2.5-flash, gemini-2.5-flash-001, gemini-2.0-flash, gemini-2.0-flash-001, gemini-1.5-flash-002, Embedding model: text-embedding-005, Open models: Gemma 2, Gemma 3, google/gemma-2b, google/gemma-7b-it, google/paligemma-3b-mix-448, PaliGemma, Qwen3:8b, Qwen/Qwen2.5-0.5B, Llama 3.1, Llama 3.2, Llama 4 Scout, PLLuM-12B-chat, Stable Diffusion XL, FLUX.1-dev, DeepSeek-R1, and xsanskarx/thinkygemma-4b, Agent and preview models: antigravity-preview-05-2026

Notebooks (29)

Agents & ADK (4)

• Deploy your containerized agent on Agent Runtime (prev. Agent Engine) — Deploys a BYOC ADK weather agent to Agent Runtime and queries it through the Vertex AI API.
• Get started with Sessions and Memory Bank for ADK agents in Cloud Run — Builds an ADK weather agent with Vertex AI Sessions, Memory Bank, and Cloud Run deployment.
• Get started with Sessions and Memory Bank for ADK agents in Google Kubernetes Engine — Builds an ADK weather agent with Vertex AI Sessions, Memory Bank, and GKE deployment.
• Intro to Managed Agents API on Agent Platform (Python) — Shows how to create, inspect, interact with, and delete Managed Agents with google-genai.

Gemini (8)

• Debugging and Optimizing Agents: A Guide to Tracing in Agent Engine — Builds, deploys, and traces a Gemini LangChain agent on Vertex AI Agent Engine.
• Deploying an Agent with Agent Engine and MCP Toolbox for Databases — Deploys a Gemini hotel-booking agent on Agent Engine using Cloud SQL, Cloud Run Toolbox, and LangGraph.
• Monitor batch prediction with Gemini API — Orchestrates and monitors Gemini batch predictions with Vertex AI Pipelines and BigQuery output.
• Evaluate generated answers from Retrieval-Augmented Generation (RAG) using Rapid Evaluation and Dataflow ML with Vertex AI pipelines — Builds a Vertex AI Pipeline to batch-evaluate RAG Q&A outputs with Rapid Eval API and Dataflow ML.
• Intro to Agent Platform Multimodal Datasets — Builds Agent Platform multimodal datasets for Gemini tuning validation, resource estimates, tuning, and batch prediction.
• Supervised Fine-Tuning with integrated Gen AI Evaluation — Fine-tunes gemini-2.5-flash with automatic Gen AI Evaluation metrics at each checkpoint.
• Vertex AI SFT Gemini Migration Recipe — Migrates explicit SFT hyperparameters from gemini-1.5-flash-002 tuning jobs to gemini-2.5-flash.
• Production & Scalable RAG Pipeline Using BigFrames — Builds a scalable BigFrames RAG pipeline over Stack Overflow data with BigQuery, Vertex AI, and LangChain.

Open Models (16)

• Accelerate LLM Inference with EAGLE Speculative Decoding on Vertex AI — Benchmarks EAGLE speculative decoding for Llama 4 Scout on Vertex AI against a baseline endpoint.
• Hugging Face DLCs: Fine-tuning Gemma with Transformer Reinforcement Learning (TRL) on Vertex AI — Fine-tunes google/gemma-2b with TRL SFT and LoRA in a Vertex AI custom container job.
• Get started with Vertex AI Model Garden SDK — Deploy and test open models on Vertex AI with the Model Garden SDK.
• Import, Deploy, and Serve custom open models on Vertex AI using Vertex AI Model Garden SDK. — Imports Hugging Face open-model weights to GCS, deploys them on Vertex AI, and serves predictions.
• Get started with Model Garden Terraform Deployment — Deploys Model Garden and Hugging Face open models to Vertex AI endpoints with Terraform.
• Serving Gemma 3 with Ollama on Cloud Run — Deploys Gemma 3 with Ollama as a private GPU-backed Cloud Run API and tests it with curl and Python.
• Running Qwen 3 with Ollama in Cloud Run for Agents — Deploys Qwen3:8b with Ollama on GPU Cloud Run and uses it as an ADK agent model.
• Serving Gemma 3 with vLLM on Cloud Run — Deploys Gemma 3 on Cloud Run with vLLM as an OpenAI-compatible chat API.
• Deploying Multiple LoRA Adapters on Vertex AI with vLLM — Deploys Gemma 2 with multiple LoRA adapters on Vertex AI using a custom vLLM container.
• Hugging Face DLCs: Serving PaliGemma using Pytorch Inference on Vertex AI with Custom Handler — Deploys gated PaliGemma from Hugging Face to Vertex AI using a PyTorch DLC and custom handler.
• Hugging Face DLCs: Serving PLLuM using Pytorch Inference on Vertex AI with Custom Handler — Deploys CYFRAGOVPL/PLLuM-12B-chat to Vertex AI with a Hugging Face PyTorch custom handler.
• Hugging Face DLCs: Serving Gemma with Text Generation Inference (TGI) on Vertex AI — Deploys gated Gemma 7B IT from Hugging Face to Vertex AI using a TGI deep learning container.
• Hugging Face DLCs: Serving Gemma 2 with multiple LoRA adapters with Text Generation Inference (TGI) on Vertex AI — Deploys Gemma 2 with SQL and code LoRA adapters on Vertex AI using a Hugging Face TGI custom handler.
• Cloud Run GPU Inference: Gemma 2 RAG Q&A with Ollama and LangChain — Deploys Gemma 2 on Cloud Run GPU with Ollama and builds a LangChain RAG Q&A chain.
• Deploying Llama 3 on Google Kubernetes Engine with Cloud Functions and vLLM — Deploys Llama 3.2 on GKE with vLLM and exposes it through a Gen2 Cloud Function.
• Serving Open-Source LLMs on Vertex AI with LiteLLM and OpenAI-Compatible APIs — Deploys a Llama 3.1 Model Garden model on Vertex AI and calls it through LiteLLM OpenAI-style APIs.

Vertex AI Search (1)

• Event-based Triggering of Manual Recrawl for Vertex AI Search Advanced Website Datastores — Automates Vertex AI Search manual recrawl from JSON URL lists uploaded to Cloud Storage.

Related concepts

• Open & Partner Models · Agents & ADK · Agent Engine · Function Calling & Tools · Tuning & Customization · Evaluation · RAG & Grounding · Applied Use Cases

---

Part of the Generative AI Google Brain · MLOps & Deployment - Best Practices