Generative AI Google Brain

MLOps & Deployment - Best Practices

13 min read

MLOps & Deployment — Best Practices

Distilled from 29 notebooks tagged MLOps & Deployment in the GoogleCloudPlatform/generative-ai repository. The From the notebooks section below cites the per-notebook source for grounding.

Do this

  • Keep project, region, model, bucket, endpoint, and runtime settings explicit and reusable through config files or environment variables.
  • Use reproducible packaging: Dockerfile, requirements.txt, setup.py, Terraform, managed SDK deployment options, or Cloud Storage model snapshots depending on the runtime.
  • Test agents, custom handlers, tool definitions, serving containers, and prediction paths locally before deploying to Agent Runtime, Agent Engine, Vertex AI, Cloud Run, or GKE.
  • Store durable session state, memory, datasets, pipeline artifacts, model artifacts, and evaluation outputs outside ephemeral runtimes using managed services such as Vertex AI Sessions, Memory Bank, Cloud Storage, BigQuery, or Vertex AI Experiments.
  • Use dedicated service accounts and grant explicit roles for Vertex AI, Artifact Registry, Cloud Storage, Cloud Build, Cloud Run, Cloud SQL, Secret Manager, Dataflow, GKE, and Discovery Engine as required.
  • Keep production services private by default with IAM authentication; avoid tutorial-only unauthenticated Cloud Run, Toolbox, or function deployments in production.
  • Check supported deployment options, EULAs, gated-model access, quotas, accelerator availability, machine compatibility, and regional constraints before deploying models.
  • Use Secret Manager or fine-grained tokens for Hugging Face and tool credentials instead of baking secrets into images or source code.
  • Poll asynchronous operations until terminal states and fail clearly when batch prediction, tuning, pipeline, Dataflow, deployment, or agent creation jobs do not succeed.
  • Use tracing, logs, Cloud Trace filters, DataFrames, sampled BigQuery outputs, row-level metrics, and aggregate summaries to debug and evaluate deployed systems.
  • For RAG and batch workflows, validate input schemas, use separate training, validation, and prediction subsets, partition large tables, deduplicate incremental outputs, and avoid loading large BigQuery data into local memory.
  • Tune serving concurrency, keep-alive settings, startup warming, autoscaling targets, GPU memory settings, and model length or LoRA limits based on accelerator capacity.
  • Use parameterized SQL, request timeouts, retry handling, JSON validation, and explicit HTTP method checks when agents or functions call external tools or services.
  • Clean up tutorial resources aggressively, including agents, endpoints, buckets, datasets, Cloud Run services, GKE clusters, Artifact Registry repositories, Cloud SQL instances, pipeline jobs, and local directories.

Avoid this

  • Forgetting to enable required APIs, billing, ADC authentication, service-agent roles, or project and region variables before running deployment code.
  • Using in-memory session storage for production agents, which loses state on shutdown and does not work reliably across scaled Cloud Run or GKE instances.
  • Deploying unauthenticated Cloud Run, Toolbox, or function services from tutorial samples without replacing them with production IAM authentication.
  • Assuming asynchronous operations are complete immediately instead of polling agent creation, batch prediction, Dataflow, tuning, deployment, pod readiness, or pipeline state.
  • Ignoring region, quota, accelerator, and machine compatibility constraints, especially for Cloud Run GPU, H100 or L4 deployments, Model Garden endpoints, and US-region BigQuery data.
  • Mishandling gated Hugging Face models by missing license acceptance, read-only tokens, Secret Manager usage, or required service-account permissions.
  • Loading large BigQuery or model artifacts into local memory or container images without considering BigFrames remote functions, Cloud Storage transfers, GCS FUSE paths, startup time, and storage limits.
  • Leaving endpoints, agents, buckets, datasets, clusters, Cloud SQL instances, Artifact Registry repositories, Cloud Functions, or Cloud Run services running after tutorials, causing unnecessary charges.

From the notebooks

Deploy your containerized agent on Agent Runtime (prev. Agent Engine)

  • Stores project, model, model region, and location settings in config.json for the containerized agent.
  • Uses a Dockerfile and requirements.txt to make the agent runtime reproducible.
  • Wraps the ADK root_agent with agent_engines.AdkApp before exposing runtime endpoints.
  • Defines both regular and streaming FastAPI endpoints for agent invocation.
  • Uses agent_framework=“google-adk” so the deployed agent can be used through the Google Cloud console playground.

Get started with Sessions and Memory Bank for ADK agents in Cloud Run

  • Store production ADK session data outside the agent runtime.
  • Use VertexAISessionService for scalable managed session storage.
  • Provide a memory tool on the Agent and a memory_service on the Runner.
  • Use a session_service on the Runner when using managed sessions.
  • Create new sessions during testing to prove long-term memory carries across sessions.

Get started with Sessions and Memory Bank for ADK agents in Google Kubernetes Engine

  • Store production ADK session data outside the agent runtime.
  • Use VertexAISessionService for scalable managed session storage.
  • Use VertexAiMemoryBankService for persistent long-term memory.
  • Provide both a memory tool on the Agent and a memory service on the Runner.
  • Provide a session service on the Runner when using managed sessions.

Intro to Managed Agents API on Agent Platform (Python)

  • Validate authentication, API enablement, service agent role, and user access before creating agents.
  • Use unique agent IDs with uuid to avoid naming collisions.
  • Poll agent creation status with client.agents.get because creation is asynchronous.
  • Delete test agents after the demo to keep the project clean.
  • Reuse environment IDs when filesystem or execution context must persist across turns.

Debugging and Optimizing Agents: A Guide to Tracing in Agent Engine

  • Enable tracing with enable_tracing=True when debugging agent execution.
  • Test the agent locally before deploying it to Agent Engine.
  • Use Cloud Trace filters such as openinference.span.kind:AGENT and root:AgentExecutor to narrow trace results.
  • Inspect traces in both the Cloud Console and the Cloud Trace Python SDK.
  • Convert spans to pandas DataFrames for programmatic trace analysis.

Deploying an Agent with Agent Engine and MCP Toolbox for Databases

  • Test the HotelBookingAgent locally before deploying it to Agent Engine.
  • Use Secret Manager to provide the Toolbox tools file to Cloud Run.
  • Grant service accounts explicit roles for Cloud SQL, Vertex AI, Secret Manager, and service usage.
  • Use parameterized SQL statements in Toolbox tool definitions.
  • Initialize Vertex AI with a Cloud Storage staging bucket before Agent Engine deployment.

Monitor batch prediction with Gemini API

  • Initialize Vertex AI with project, location, and staging_bucket before running the pipeline.
  • Use timestamped BigQuery output table names to avoid collisions across batch runs.
  • Wrap the batch job in a Vertex AI Pipeline and use VertexNotificationEmailOp because Gemini batch prediction lacks built-in completion notifications.
  • Poll BatchPredictionJob until has_ended and fail the component when has_succeeded is false.
  • Sample prediction results from BigQuery instead of loading the whole output table.

Evaluate generated answers from Retrieval-Augmented Generation (RAG) using Rapid Evaluation and Dataflow ML with Vertex AI pipelines

  • Validate the evaluation dataset before starting remote evaluation.
  • Store pipeline data, source modules, requirements, outputs, and temporary files in Cloud Storage paths.
  • Package the Apache Beam module with requirements.txt and setup.py for Dataflow workers.
  • Use WaitGcpResourcesOp after DataflowPythonJobOp before reading output artifacts.
  • Retrieve both row-level metrics and aggregated summary metrics after pipeline completion.

Intro to Agent Platform Multimodal Datasets

  • Use BigFrames instead of pandas for larger datasets.
  • Inspect created datasets with resource name, display name, BigQuery URI, and BigFrames preview.
  • Attach the read configuration to the dataset before tuning and batch prediction workflows.
  • Run tuning validity assessment before starting supervised fine-tuning.
  • Estimate tuning resources before running a tuning job.

Supervised Fine-Tuning with integrated Gen AI Evaluation

  • Use separate training and validation JSONL datasets for supervised tuning with integrated evaluation.
  • Store detailed row-level evaluation results in Cloud Storage using output_config.
  • Use custom model-based metrics with clear prompt templates and optional judge model system instructions.
  • Monitor asynchronous tuning jobs by refreshing job state until terminal completion.
  • Inspect checkpoint evaluation runs in Vertex AI Experiments to compare model progress over time.

Vertex AI SFT Gemini Migration Recipe

  • Do not apply the same hyperparameters from legacy Gemini models to latest Gemini models because model architecture and tuning infrastructure changed.
  • Use API or SDK defaults for new models when the legacy tuning job did not explicitly set hyperparameters.
  • Reuse the existing training and validation dataset URIs when creating the migrated tuning job.
  • Check tuning data statistics before changing epoch count for migration.

Production & Scalable RAG Pipeline Using BigFrames

  • Use BigFrames to process BigQuery data with pandas-like syntax without moving terabyte-scale data out of BigQuery.
  • Parameterize scheduled runs with RUN_DATE, IS_INCREMENTAL, LOOK_BACK_DAYS, START_DATE, and END_DATE.
  • Sort by last_edit_date and drop duplicate question_id values before embedding.
  • Convert HTML questions and answers to Markdown before chunking and generation.
  • Use chunk overlap and consider preserving paragraphs, sections, markdown hierarchy, code blocks, lists, and question-answer grouping in production.

Accelerate LLM Inference with EAGLE Speculative Decoding on Vertex AI

  • Pin package versions for reproducibility.
  • Deploy baseline and EAGLE endpoints on identical hardware for a fair comparison.
  • Run 100-prompt smoke tests before the full benchmark.
  • Use ShareGPT conversations as a realistic benchmark workload.
  • Use the model tokenizer locally so prompt and response token lengths are measured accurately.

Hugging Face DLCs: Fine-tuning Gemma with Transformer Reinforcement Learning (TRL) on Vertex AI

  • Use a Hugging Face PyTorch training DLC instead of manually building the training environment.
  • Use LoRA and an 8-bit optimizer to reduce VRAM requirements for fine-tuning.
  • Set output_dir under the /gcs mounted bucket path so trainer outputs are uploaded to Cloud Storage.
  • Pass logging environment variables so container logs stream cleanly to Google Cloud Logs Explorer.
  • Use a read-only or fine-grained Hugging Face token for gated model access.

Get started with Vertex AI Model Garden SDK

  • Use list_deployable_models before selecting a model ID.
  • Use list_deploy_options to verify supported deployment configurations before deploying.
  • Use environment variables for project and region defaults.
  • Accept model EULAs explicitly when required.
  • Wrap deployment attempts in try/except to surface deployment errors.

Import, Deploy, and Serve custom open models on Vertex AI using Vertex AI Model Garden SDK.

  • Use environment variables as fallbacks for project and region.
  • Enable hf_transfer for faster Hugging Face downloads.
  • Upload large model files to Cloud Storage with parallel chunk transfers.
  • Call list_deploy_options before deployment to verify supported configurations and resource needs.
  • Use a dedicated endpoint for prediction calls.

Get started with Model Garden Terraform Deployment

  • Store Terraform configurations in Git.
  • Use remote state such as Google Cloud Storage for team environments.
  • Separate dev, staging, and prod environments.
  • Use variables for reusable configuration.
  • Add labels to resources for organization and cost tracking.

Serving Gemma 3 with Ollama on Cloud Run

  • Keep the Cloud Run service private with —no-allow-unauthenticated and use IAM authentication.
  • Use a dedicated service account for the Cloud Run service.
  • Embed model weights in the container image with ollama pull during build.
  • Set OLLAMA_KEEP_ALIVE=-1 so model weights are not unloaded from GPU.
  • Set Cloud Run concurrency to match OLLAMA_NUM_PARALLEL and tune it for GPU utilization.

Running Qwen 3 with Ollama in Cloud Run for Agents

  • Keep the Cloud Run service private with —no-allow-unauthenticated and use IAM authentication.
  • Use a dedicated service account for the Cloud Run service.
  • Set OLLAMA_KEEP_ALIVE=-1 so model weights are not unloaded from GPU memory.
  • Warm the model at startup with a dummy ollama run request.
  • Set temperature=0.1 for stable function calling.

Serving Gemma 3 with vLLM on Cloud Run

  • Use Secret Manager to pass the Hugging Face token to Cloud Build instead of baking it into code.
  • Download model weights at image build time and set HF_HUB_OFFLINE=1 for cached runtime loading.
  • Keep the Cloud Run service private and rely on IAM authentication.
  • Set max instances according to available NVIDIA L4 GPU quota.
  • Use Artifact Registry for the built serving image.

Deploying Multiple LoRA Adapters on Vertex AI with vLLM

  • Store model files in Cloud Storage instead of baking them into a large Docker image.
  • Enable parallel composite uploads for faster Cloud Storage transfers.
  • Use a startup entrypoint that fails fast when GCS downloads fail before starting vLLM.
  • Set temperature to 0.0 for deterministic SQL generation.
  • Configure Vertex AI health and predict routes explicitly for the serving container.

Hugging Face DLCs: Serving PaliGemma using Pytorch Inference on Vertex AI with Custom Handler

  • Test the handler logic locally before packaging it as handler.py.
  • Use HF_HOME to control the local Hugging Face cache location.
  • Upload large model files to Cloud Storage with gsutil parallel composite uploads.
  • Validate request instances contain both prompt and image_base64.
  • Use torch.inference_mode() for generation.

Hugging Face DLCs: Serving PLLuM using Pytorch Inference on Vertex AI with Custom Handler

  • Test the custom handler logic locally before packaging it as handler.py.
  • Use HF_HOME to keep Hugging Face cache artifacts in the tutorial directory.
  • Copy the model snapshot and handler.py to Cloud Storage before Vertex AI upload.
  • Use gsutil -m and parallel composite upload settings for large model directories.
  • Validate the serving container locally with LocalModel before deploying to Vertex AI.

Hugging Face DLCs: Serving Gemma with Text Generation Inference (TGI) on Vertex AI

  • Use environment variables for PROJECT_ID and GOOGLE_CLOUD_REGION with us-central1 as the default region.
  • Use a read-only or fine-grained Hugging Face token for google/gemma-7b-it.
  • Set TGI runtime variables such as NUM_SHARD, MAX_INPUT_TOKENS, MAX_TOTAL_TOKENS, and MAX_BATCH_PREFILL_TOKENS.
  • Apply the tokenizer chat template before sending prompts to an instruction-following model.
  • Undeploy endpoints, delete endpoints, and delete registered models to avoid unnecessary costs.

Hugging Face DLCs: Serving Gemma 2 with multiple LoRA adapters with Text Generation Inference (TGI) on Vertex AI

  • Test the custom handler logic locally before packaging handler.py for serving.
  • Use response_mime_type application/json and a response_schema for the Gemini router output.
  • Validate that each prediction instance contains the inputs key.
  • Use gsutil -m and parallel_composite_upload_threshold=150M for large model artifact uploads.
  • Store handler.py and requirements.txt with the model artifacts in Cloud Storage.

Cloud Run GPU Inference: Gemma 2 RAG Q&A with Ollama and LangChain

  • Store Gemma 2 9B and similarly sized model weights directly in the container image for startup time and scalability.
  • Consider storage requirements before placing larger model weights in the image.
  • Use e2-highcpu-32 for Cloud Build to speed up parallel downloads.
  • Set OLLAMA_KEEP_ALIVE=-1 to avoid unloading model weights from GPU memory.
  • Use a dummy startup request to load the model into GPU memory.

Deploying Llama 3 on Google Kubernetes Engine with Cloud Functions and vLLM

  • Uses private GKE nodes and an internal load balancer for the model service.
  • Separates the model server from the API layer by putting Cloud Functions in front of GKE.
  • Uses a VPC connector so the Cloud Function can reach the private GKE endpoint.
  • Passes the internal endpoint IP through an environment variable instead of hardcoding it in function code.
  • Creates a dedicated service account for the Cloud Function and grants required IAM roles.

Serving Open-Source LLMs on Vertex AI with LiteLLM and OpenAI-Compatible APIs

  • Check list_deploy_options() before deployment to verify supported configurations and resource needs.
  • Use environment variables for project and region defaults.
  • Validate the endpoint resource name before configuring LiteLLM.
  • Keep backend or external API execution separate from model tool-call argument generation.
  • Append assistant tool calls and tool responses to the message history before the second model call.

Event-based Triggering of Manual Recrawl for Vertex AI Search Advanced Website Datastores

  • Use a Cloud Storage staging bucket for JSON files containing URLs to recrawl.
  • Trigger processing on google.cloud.storage.object.v1.finalized events and filter for .json files.
  • Use functions-framework, described as the recommended way at the time of the notebook.
  • Fetch an access token with google-auth default credentials before calling the REST API.
  • Set a request timeout and retry connection or timeout failures up to three times.

Back to MLOps & Deployment · Best Practices Map

Graph View

Backlinks