Generative AI Google Brain

open-models-use-cases-deploy-llama3-vllm-gke-cloud-function

4 min read

Deploying Llama 3 on Google Kubernetes Engine with Cloud Functions and vLLM

Source notebook

Repo path: open-models/use-cases/deploy_llama3_vllm_gke_cloud_function.ipynb · Open on GitHub · advanced

Deploys Llama 3.2 on GKE with vLLM and exposes it through a Gen2 Cloud Function.

Summary

The notebook teaches how to create a private Autopilot GKE cluster, deploy a selected Llama 3.2 model with vLLM on GPUs, and test the internal load balancer endpoint. It then builds and deploys a Python Gen2 Cloud Function that forwards HTTP POST prompts to the vLLM /generate endpoint through a VPC connector. The workflow ends with cleanup commands for the Kubernetes deployment, GKE cluster, Cloud Function, and VPC connector.

Key code patterns

Create private Autopilot cluster

gcloud beta container clusters create-auto CLUSTER_NAME \
  --region REGION \
  --enable-private-nodes \
  --network NETWORK_URI \
  --subnetwork SUBNET_URI

Creates the GKE environment that hosts the GPU-backed vLLM server.

Configure vLLM args

ARGS_TEMPLATE = """args:
  - python
  - -m
  - vllm.entrypoints.api_server
  - --host 0.0.0.0
  - --port 7080
  - --model=gs://vertex-model-garden-public-us/llama3.2/{}
  - --tensor-parallel-size {}
  - --gpu-memory-utilization 0.95"""

Builds the container command that serves a Llama 3.2 model from Cloud Storage with vLLM.

Apply Kubernetes manifest

with open("llama_32.yaml", "w") as f:
    f.write(K8S_YAML)
 
! kubectl apply -f llama_32.yaml

Writes the generated deployment and service YAML, then deploys it to the cluster.

Find internal load balancer IP

command = [
  "kubectl", "get", "service", service_name,
  "-n", namespace,
  "-o", "jsonpath={.status.loadBalancer.ingress[0].ip}",
]
result = subprocess.run(command, capture_output=True, text=True, check=True)

Retrieves the private endpoint that the Cloud Function later calls through VPC access.

Call vLLM endpoint

request_payload = {
  "prompt": user_message,
  "max_tokens": max_tokens,
  "temperature": temperature,
}
response = requests.post(endpoint_url, headers=headers, json=request_payload, timeout=120)

Shows the request shape used for direct testing and for the Cloud Function proxy.

HTTP Cloud Function proxy

@functions_framework.http
def call_llama_service(request):
    prompt = request.get_json()["prompt"]
    endpoint_url = f"http://{LLAMA_ILB_IP}:{SERVICE_PORT}/generate"
    response = requests.post(endpoint_url, json={"prompt": prompt})
    return ({"prediction": response.json()}, 200)

Exposes the private GKE-hosted model through a controlled HTTP-triggered function.

Deploy Gen2 function with VPC connector

gcloud functions deploy FUNCTION_NAME \
  --gen2 --runtime python311 \
  --entry-point call_llama_service \
  --trigger-http --no-allow-unauthenticated \
  --set-env-vars LLAMA_ENDPOINT_IP=ILB_IP \
  --vpc-connector VPC_CONNECTOR_NAME

Connects the serverless API layer to the private GKE service endpoint.

Models & APIs used

When to use this

Use this pattern when you need to serve open Llama 3.2 models on GPU-backed GKE and expose them through a private, serverless HTTP API.

Gotchas & caveats

  • Billing must be enabled for the Google Cloud project.
  • PROJECT_ID, NETWORK_NAME, SUBNET_NAME, CLUSTER_NAME, SERVICE_ACCOUNT_NAME, and VPC_CONNECTOR_NAME must be provided.
  • The subnet and network must be in the same region as the cluster and VPC connector.
  • The notebook installs kubectl through gcloud components before applying Kubernetes resources.
  • Model deployment is expected to take 5 to 15 minutes, and larger models may take longer.
  • The code waits up to 600 seconds for pod readiness and server logs before warning that deployment took longer than expected.
  • The Cloud Function depends on LLAMA_ENDPOINT_IP being set to the internal load balancer IP.
  • The function blocks non-POST requests and requires a JSON body with a prompt key.
  • The function is deployed with —no-allow-unauthenticated, so callers need authorization.
  • Cleanup is needed to avoid continuous charges for GKE, Cloud Functions, and VPC connector resources.

Best practices

  • Uses private GKE nodes and an internal load balancer for the model service.
  • Separates the model server from the API layer by putting Cloud Functions in front of GKE.
  • Uses a VPC connector so the Cloud Function can reach the private GKE endpoint.
  • Passes the internal endpoint IP through an environment variable instead of hardcoding it in function code.
  • Creates a dedicated service account for the Cloud Function and grants required IAM roles.
  • Sets request timeouts when calling the vLLM service.
  • Validates HTTP method and request JSON before forwarding prompts.
  • Provides explicit cleanup commands to reduce unnecessary charges.

Graph View

Backlinks