Generative AI Google Brain

open-models-serving-cloud-run-vllm-gemma3-inference

3 min read

Serving Gemma 3 with vLLM on Cloud Run

Source notebook

Repo path: open-models/serving/cloud_run_vllm_gemma3_inference.ipynb · Open on GitHub · advanced

Deploys Gemma 3 on Cloud Run with vLLM as an OpenAI-compatible chat API.

Summary

The notebook shows how to package a gated Hugging Face Gemma 3 model into a vLLM container and deploy it to Cloud Run with an NVIDIA L4 GPU. It walks through Google Cloud and Hugging Face authentication, Secret Manager-backed Cloud Build, Artifact Registry image publishing, private Cloud Run deployment, and testing via curl and the OpenAI Python client.

Key code patterns

Model and service naming

MODEL = "google/gemma-3-4b-it"
PROJECT_ID = "[your-project-id]"
REGION = "us-central1"
SERVICE_NAME = f"vllm--{MODEL.replace('.', '-').replace('/','--')}"

Creates a Cloud Run-safe service name from the selected Hugging Face model id.

Hugging Face token login

from huggingface_hub import get_token, interpreter_login
 
interpreter_login(new_session=False)
HF_TOKEN = get_token()

Authenticates to download the gated Gemma model after license acceptance.

Offline vLLM container

FROM us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/pytorch-vllm-serve:20250312_0916_RC01
ARG MODEL
ENV HF_MODEL=$MODEL
ENV HF_HOME=/model-cache
RUN --mount=type=secret,id=HF_TOKEN HF_TOKEN=$(cat /run/secrets/HF_TOKEN) huggingface-cli download ${HF_MODEL}
ENV HF_HUB_OFFLINE=1

Downloads model weights during build with a secret and forces runtime use of the cached model.

OpenAI-compatible vLLM server

ENTRYPOINT python3 -m vllm.entrypoints.openai.api_server \
    --port ${PORT:-8080} \
    --model ${HF_MODEL} \
    --max-num-seqs=4 \
    --max-model-len 32768

Starts vLLM using its OpenAI-compatible API server inside Cloud Run.

GPU Cloud Run deployment

gcloud beta run deploy $SERVICE_NAME \
    --gpu=1 --gpu-type=nvidia-l4 \
    --cpu=8 --memory=32Gi \
    --no-allow-unauthenticated \
    --no-cpu-throttling \
    --max-instances ${MAX_INSTANCES}

Deploys a private GPU-backed Cloud Run service sized for vLLM inference.

OpenAI Python client test

client = OpenAI(api_key="EMPTY", base_url=f"{service_url}/v1")
chat_response = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": [{"type": "text", "text": prompt}, {"type": "image_url", "image_url": {"url": image_url}}]}],
    temperature=0.5,
    extra_headers=f"Bearer {identity_token}",
)

Tests the private Cloud Run endpoint through the OpenAI-compatible client interface.

Models & APIs used

  • Models: google/gemma-3-4b-it
  • APIs / services: Cloud Run, Cloud Build, Secret Manager, Artifact Registry, IAM, Hugging Face Hub
  • SDKs / libraries: huggingface_hub, openai, vLLM, Google Cloud SDK

When to use this

Use this pattern to serve a Hugging Face Gemma 3 model as a private OpenAI-compatible API on GPU-backed Cloud Run.

Gotchas & caveats

  • Cloud Run GPU support is in preview in the notebook.
  • The project needs Total Nvidia L4 GPU allocation per project per region quota for Cloud Run.
  • Gemma models on Hugging Face are gated and require license acceptance plus a new read-only access token after acceptance.
  • The deployment script can take 15-45 minutes to finish.
  • Cloud Run GPU deployment requires —no-cpu-throttling.
  • —max-instances must be equal to or lower than the project’s NVIDIA L4 GPU quota.
  • The notebook keeps the service private with —no-allow-unauthenticated and uses identity tokens for requests.
  • The cleanup cell references $LOCATION even though the notebook uses REGION.

Best practices

  • Use Secret Manager to pass the Hugging Face token to Cloud Build instead of baking it into code.
  • Download model weights at image build time and set HF_HUB_OFFLINE=1 for cached runtime loading.
  • Keep the Cloud Run service private and rely on IAM authentication.
  • Set max instances according to available NVIDIA L4 GPU quota.
  • Use Artifact Registry for the built serving image.
  • Create a dedicated Cloud Run service account for the deployed service.

Graph View

Backlinks