Generative AI Google Brain

open-models-get-started-with-model-garden-terraform-deployment

3 min read

Get started with Model Garden Terraform Deployment

Source notebook

Repo path: open-models/get_started_with_model_garden_terraform_deployment.ipynb · Open on GitHub · intermediate

Deploys Model Garden and Hugging Face open models to Vertex AI endpoints with Terraform.

Summary

This notebook teaches how to use Terraform as infrastructure as code for Vertex AI Model Garden deployments. It walks through project setup, Terraform configuration, model discovery, endpoint deployment, prediction calls, advanced compute settings, gated Hugging Face models, and cleanup.

Key code patterns

Initialize Vertex AI

PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")
LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "us-central1")
vertexai.init(project=PROJECT_ID, location=LOCATION)

Sets project and region context before deploying or calling Vertex AI resources.

Terraform Model Garden deployment

resource "google_vertex_ai_endpoint_with_model_garden_deployment" "gemma_deployment" {
  publisher_model_name = "publishers/google/models/gemma3@gemma-3-1b-it"
  location             = var.region
 
  model_config {
    accept_eula = true
  }
}

Defines a repeatable Vertex AI endpoint deployment for a curated Model Garden model.

Dedicated endpoint prediction

endpoint = aiplatform.Endpoint(ENDPOINT_ID)
response = endpoint.predict(
    instances=[{"prompt": "Tell me a joke about AI", "temperature": 0.7, "max_tokens": 35}],
    use_dedicated_endpoint=True,
)

Shows how to call the deployed endpoint with the Vertex AI SDK.

OpenAI-compatible endpoint call

endpoint_url = f"https://{endpoint.gca_resource.dedicated_endpoint_dns}/v1beta1/{endpoint.resource_name}"
client = openai.OpenAI(base_url=endpoint_url, api_key=creds.token)
response = client.chat.completions.create(model="", messages=[{"role": "user", "content": "Tell me a joke about AI"}])

Uses Google credentials with the OpenAI SDK for compatible deployed models.

Hugging Face gated model

variable "hugging_face_token" {
  type      = string
  sensitive = true
}
model_config {
  accept_eula = true
  hugging_face_access_token = var.hugging_face_token
}

Passes a Hugging Face access token for gated model deployment.

Models & APIs used

  • Models: publishers/google/models/gemma3@gemma-3-1b-it, Qwen/Qwen2.5-0.5B, publishers/google/models/paligemma@paligemma-224-float32, meta-llama/Llama-3.2-1B
  • APIs / services: Vertex AI, Vertex AI Model Garden, Hugging Face Hub
  • SDKs / libraries: google-cloud-aiplatform, vertexai, openai, google-auth, requests, huggingface_hub

When to use this

Use this pattern when you need repeatable Terraform-managed Vertex AI deployments for Model Garden or Hugging Face open models.

Gotchas & caveats

  • Requires a Google Cloud project with billing enabled.
  • Vertex AI API must be enabled.
  • Requires sufficient IAM permissions such as Vertex AI Administrator or Editor.
  • Terraform must be installed separately in the notebook environment.
  • Deployments can take 10-15 minutes, or 20-30 minutes for multiple models.
  • Custom GPU deployments require sufficient quota for resources such as g2-standard-16 and NVIDIA_L4.
  • Gated Hugging Face models require a Hugging Face access token.
  • Cleanup is needed to avoid unnecessary charges.

Best practices

  • Store Terraform configurations in Git.
  • Use remote state such as Google Cloud Storage for team environments.
  • Separate dev, staging, and prod environments.
  • Use variables for reusable configuration.
  • Add labels to resources for organization and cost tracking.
  • Run terraform plan before terraform apply.
  • Use environment variables or secret management tools for sensitive data.

Graph View

Backlinks