Generative AI Google Brain

open-models-serving-cloud-run-ollama-gemma3-inference

3 min read

Serving Gemma 3 with Ollama on Cloud Run

Source notebook

Repo path: open-models/serving/cloud_run_ollama_gemma3_inference.ipynb · Open on GitHub · intermediate

Deploys Gemma 3 with Ollama as a private GPU-backed Cloud Run API and tests it with curl and Python.

Summary

This notebook teaches how to package a Gemma 3 Ollama model into a custom container, build it with Cloud Build, push it to Artifact Registry, and deploy it to Cloud Run with an NVIDIA L4 GPU. It demonstrates enabling required APIs, creating a service account and repository, deploying a private Cloud Run service, then calling the deployed Ollama API with curl and the Ollama Python client.

Key code patterns

model and service naming

MODEL = "gemma3:4b"
PROJECT_ID = "[your-project-id]"
REGION = "us-central1"
MODEL_NAME_ESCAPED = MODEL.translate(str.maketrans(".:/", "---"))
SERVICE_NAME = f"ollama--{MODEL_NAME_ESCAPED}"

Creates a Cloud Run-safe service name from the selected Ollama model id.

Ollama container with embedded model

FROM ollama/ollama:0.6.0
ARG MODEL
ENV MODEL=$MODEL
ENV OLLAMA_HOST 0.0.0.0:8080
ENV OLLAMA_MODELS /models
ENV OLLAMA_KEEP_ALIVE -1
RUN ollama serve & sleep 5 && ollama pull $MODEL
CMD ["-c", "ollama serve  & (ollama run $MODEL 'Say one word' &) && wait"]

Builds a container that downloads the model at image build time and warms it on startup.

Cloud Run GPU deploy

gcloud beta run deploy $SERVICE_NAME \
  --image=${REGION}-docker.pkg.dev/$PROJECT_ID/$AR_REPO_NAME/$SERVICE_NAME \
  --gpu=1 --gpu-type=nvidia-l4 \
  --cpu=8 --memory=32Gi \
  --concurrency 4 \
  --set-env-vars OLLAMA_NUM_PARALLEL=4 \
  --no-allow-unauthenticated \
  --no-cpu-throttling

Deploys the Ollama service with GPU, concurrency, private access, and CPU throttling disabled.

authenticated curl request

SERVICE_URL=$(gcloud run services describe ${SERVICE_NAME} --format 'value(status.url)')
AUTH_TOKEN=$(gcloud auth print-identity-token -q)
curl -s -X POST "${SERVICE_URL}/api/generate" \
-H "Authorization: Bearer ${AUTH_TOKEN}" \
-H "Content-Type: application/json" \
-d '{ "model": "'${MODEL}'", "prompt": "Hello!", "stream": false}'

Shows how to call the private Cloud Run Ollama endpoint with an identity token.

Ollama Python client

from ollama import Client
client = Client(host=service_url, headers={"Authorization": f"Bearer {identity_token}"})
stream = client.chat(
    model=MODEL,
    messages=[{"role": "user", "content": "Why is the sky blue?"}],
    stream=True,
)

Uses the Ollama Python library against the Cloud Run-hosted endpoint.

Models & APIs used

  • Models: gemma3:4b
  • APIs / services: Cloud Run, Cloud Build, Artifact Registry, IAM
  • SDKs / libraries: gcloud, ollama

When to use this

Use this pattern when you need a private, serverless, GPU-backed Ollama endpoint for Gemma 3 or another Ollama-supported LLM on Cloud Run.

Gotchas & caveats

  • Google Cloud SDK must be installed or available in Colab or Vertex AI Workbench.
  • PROJECT_ID must be set before running the notebook.
  • Cloud Run GPU support is in preview.
  • The project and region need Total Nvidia L4 GPU allocation quota under Cloud Run.
  • —no-cpu-throttling is required for enabling GPU.
  • —max-instances must be equal to or lower than the project’s NVIDIA L4 GPU quota.
  • The notebook says the deploy script may take 10-45 minutes.
  • The cleanup cell uses $LOCATION even though the notebook defines REGION.

Best practices

  • Keep the Cloud Run service private with —no-allow-unauthenticated and use IAM authentication.
  • Use a dedicated service account for the Cloud Run service.
  • Embed model weights in the container image with ollama pull during build.
  • Set OLLAMA_KEEP_ALIVE=-1 so model weights are not unloaded from GPU.
  • Set Cloud Run concurrency to match OLLAMA_NUM_PARALLEL and tune it for GPU utilization.
  • Create Artifact Registry and service account only if they do not already exist.
  • Warm the model at startup with a dummy Ollama request.

Graph View

Backlinks