Generative AI Google Brain

open-models-get-started-with-model-garden-sdk

3 min read

Get started with Vertex AI Model Garden SDK

Source notebook

Repo path: open-models/get_started_with_model_garden_sdk.ipynb · Open on GitHub · intermediate

Deploy and test open models on Vertex AI with the Model Garden SDK.

Summary

This notebook teaches how to discover deployable open models in Vertex AI Model Garden, inspect deploy options, deploy models to Vertex AI endpoints, and run predictions. It demonstrates prediction through both the Vertex AI SDK endpoint interface and an OpenAI-compatible Chat Completions client. It also shows advanced deployment parameters, image generation with Stable Diffusion XL, cleanup, and common deployment error cases.

Key code patterns

Initialize Vertex AI

PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")
LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "us-central1")
vertexai.init(project=PROJECT_ID, location=LOCATION)

Sets the project and region before using Model Garden deployment APIs.

List deployable models

model_garden_models = model_garden.list_deployable_models(
    model_filter="gemma",
    list_hf_models=False,
)
deployable_models = model_garden.list_deployable_models(
    model_filter="gemma",
    list_hf_models=True,
)

Discovers Model Garden models and optionally includes Hugging Face Gallery models.

Inspect and deploy model

model_id = "google/gemma3@gemma-3-1b-it"
gemma_model = model_garden.OpenModel(model_id)
deploy_options = gemma_model.list_deploy_options(concise=True)
gemma_endpoint = gemma_model.deploy(accept_eula=True)

Uses OpenModel to check verified deployment configurations and deploy with EULA acceptance.

Predict with Vertex AI endpoint

prediction = gemma_endpoint.predict(
    instances=[{"prompt": "Tell me a joke", "temperature": 0.7, "max_tokens": 50}]
)
print(prediction.predictions[0])

Runs inference directly through the deployed Vertex AI endpoint.

Use OpenAI-compatible client

creds, project = google.auth.default()
creds.refresh(google.auth.transport.requests.Request())
url = f"https://{LOCATION}-aiplatform.googleapis.com/v1beta1/{gemma_endpoint.resource_name}"
client = openai.OpenAI(base_url=url, api_key=creds.token)
prediction = client.chat.completions.create(model="", messages=[{"role":"user","content":"Tell me a joke"}])

Shows how to call the deployed Vertex AI endpoint through the OpenAI SDK.

Advanced deployment settings

sd_endpoint = sd_model.deploy(
    machine_type="g2-standard-4",
    accelerator_type="NVIDIA_L4",
    accelerator_count=1,
    min_replica_count=1,
    max_replica_count=1,
)

Configures compute, accelerator, replica, endpoint, model, and timeout deployment parameters.

Handle deployment errors

try:
    model = model_garden.OpenModel("black-forest-labs/FLUX.1-dev")
    endpoint = model.deploy(hugging_face_access_token="invalid-token")
except Exception as e:
    print(f"Error: {e}")

Captures exceptions for invalid models, malformed names, quota, policy, EULA, and gated model access.

Models & APIs used

  • Models: google/gemma3@gemma-3-1b-it, stabilityai/stable-diffusion-xl-base-1.0, google/some-model@some-version, publisher/google/some-model@some-version, publishers/deepseek-ai/models/deepseek-r1@deepseek-r1, publishers/meta/models/llama3-2@llama-3.2-90b-vision, black-forest-labs/FLUX.1-dev
  • APIs / services: Vertex AI, Vertex AI Model Garden, Hugging Face Hub, OpenAI Chat Completions API
  • SDKs / libraries: google-cloud-aiplatform, vertexai, openai, google-auth, requests, Pillow, matplotlib

When to use this

Use this pattern when deploying open or Hugging Face models from Vertex AI Model Garden to managed Vertex AI endpoints.

Gotchas & caveats

  • Vertex AI API must be enabled for the Google Cloud project.
  • Colab requires explicit user authentication.
  • Deployment location defaults to us-central1 if GOOGLE_CLOUD_REGION is unset.
  • Some models require accept_eula=True before deployment.
  • GPU, CPU, memory, Vertex AI, or regional quotas can block deployment.
  • Organization policy constraints such as constraints/vertexai.allowedModels can block model access.
  • Hugging Face gated models require license acceptance and a valid read-only access token.
  • Deployments can take minutes, and the Stable Diffusion example sets a 3 hour timeout.

Best practices

  • Use list_deployable_models before selecting a model ID.
  • Use list_deploy_options to verify supported deployment configurations before deploying.
  • Use environment variables for project and region defaults.
  • Accept model EULAs explicitly when required.
  • Wrap deployment attempts in try/except to surface deployment errors.
  • Delete endpoints with force=True during cleanup to avoid leaving resources running.

Graph View

Backlinks