Generative AI Google Brain

open-models-serving-cloud-run-ollama-qwen3-inference

3 min read

Running Qwen 3 with Ollama in Cloud Run for Agents

Source notebook

Repo path: open-models/serving/cloud_run_ollama_qwen3_inference.ipynb · Open on GitHub · advanced

Deploys Qwen3:8b with Ollama on GPU Cloud Run and uses it as an ADK agent model.

Summary

The notebook shows how to build an Ollama container with Qwen3:8b embedded, deploy it as a private GPU-backed Cloud Run service, and test it with curl. It then retrieves a Cloud Run identity token, configures LiteLlm against the service’s OpenAI-compatible /v1 endpoint, and runs a Google ADK agent with a Python time tool.

Key code patterns

Model and service naming

MODEL = "qwen3:8b"
PROJECT_ID = "[your-project-id]"
REGION = "us-central1"
MODEL_NAME_ESCAPED = MODEL.translate(str.maketrans(".:/", "---"))
SERVICE_NAME = f"ollama--{MODEL_NAME_ESCAPED}"

Derives a Cloud Run-safe service name from the Ollama model id.

Ollama container preload

FROM ollama/ollama:latest
ARG _MODEL
ENV OLLAMA_HOST 0.0.0.0:8080
ENV OLLAMA_MODELS /models
ENV OLLAMA_KEEP_ALIVE -1
RUN ollama serve & sleep 5 && ollama pull $_MODEL
CMD ["-c", "ollama serve  & (ollama run $_MODEL 'Say one word' &) && wait"]

Pulls model weights at build time and warms the model at startup for GPU serving.

GPU Cloud Run deploy

gcloud beta run deploy $SERVICE_NAME \
  --source . \
  --cpu=8 --memory=32Gi \
  --gpu=1 --gpu-type=nvidia-l4 \
  --concurrency 4 \
  --set-env-vars OLLAMA_NUM_PARALLEL=4 \
  --no-allow-unauthenticated \
  --no-cpu-throttling \
  --no-gpu-zonal-redundancy

Deploys a private Cloud Run service with one NVIDIA L4 GPU and Ollama parallelism aligned to concurrency.

Authenticated Ollama request

AUTH_TOKEN=$(gcloud auth print-identity-token -q)
curl -s -X POST "${SERVICE_URL}/api/generate" \
-H "Authorization: Bearer ${AUTH_TOKEN}" \
-H "Content-Type: application/json" \
-d '{ "model": "'${MODEL}'", "prompt": "Hello!", "stream": false}'

Calls the private Cloud Run service using an identity token in the Authorization header.

ADK agent over Cloud Run

root_agent = Agent(
    name="time_agent",
    model=LiteLlm(
        model=f"openai/{MODEL}",
        api_base=service_url + "/v1",
        api_key=auth_id_token,
        temperature=0.1,
    ),
    tools=[get_current_time],
)

Connects ADK to the deployed Ollama service through LiteLlm’s OpenAI-compatible interface.

Models & APIs used

  • Models: qwen3:8b
  • APIs / services: Cloud Run, Cloud Build, Artifact Registry, IAM
  • SDKs / libraries: google-genai, google-adk, litellm

When to use this

Use this pattern to serve a lightweight Ollama-hosted open model on private GPU Cloud Run and plug it into ADK agents.

Gotchas & caveats

  • Google Cloud CLI must be installed unless running in Colab or Vertex AI Workbench.
  • The project needs Total Nvidia L4 GPU allocation without zonal redundancy in the selected region.
  • Deployment may take 10-45 minutes.
  • Cloud Run GPU requires —no-cpu-throttling.
  • The private Cloud Run service requires an identity token on every request.
  • —max-instances must be equal to or lower than the project’s NVIDIA L4 GPU quota.
  • The notebook sets OLLAMA_API_BASE because it is still required for LiteLlm to work.
  • The cleanup cell uses $LOCATION even though the notebook defines REGION.

Best practices

  • Keep the Cloud Run service private with —no-allow-unauthenticated and use IAM authentication.
  • Use a dedicated service account for the Cloud Run service.
  • Set OLLAMA_KEEP_ALIVE=-1 so model weights are not unloaded from GPU memory.
  • Warm the model at startup with a dummy ollama run request.
  • Set temperature=0.1 for stable function calling.
  • Use /no_think in the agent instruction for Qwen 3 to run faster.
  • Align —concurrency with OLLAMA_NUM_PARALLEL and tune concurrency for GPU utilization.

Graph View

Backlinks