Generative AI Google Brain

open-models-get-started-with-model-garden-sdk-custom-import

3 min read

Import, Deploy, and Serve custom open models on Vertex AI using Vertex AI Model Garden SDK.

Source notebook

Repo path: open-models/get_started_with_model_garden_sdk_custom_import.ipynb · Open on GitHub · intermediate

Imports Hugging Face open-model weights to GCS, deploys them on Vertex AI, and serves predictions.

Summary

This notebook teaches how to import an open-source Hugging Face model into Vertex AI using the Vertex AI Model Garden SDK custom model import flow. It transfers model artifacts to Cloud Storage, creates a Model Garden CustomModel from the GCS URI, lists deploy options, deploys to a Vertex AI Endpoint, and runs inference with both endpoint.predict and the OpenAI ChatCompletion API.

Key code patterns

Vertex AI setup

PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")
LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "us-central1")
BUCKET_URI = f"gs://{BUCKET_NAME}"
!gcloud storage buckets create {BUCKET_URI} --project={PROJECT_ID} --location={LOCATION}
vertexai.init(project=PROJECT_ID, location=LOCATION)

Initializes project, region, Cloud Storage staging, and Vertex AI before import and deployment.

Hugging Face to GCS transfer

os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
snapshot_download(repo_id=model_id, local_dir=str(local_dir), ignore_patterns=exclude_patterns, resume_download=True)
client = storage.Client()
bucket = client.bucket(bucket_name)
transfer_manager.upload_chunks_concurrently(str(file_path), blob, chunk_size=chunk_size, max_workers=workers)

Stages model files locally, accelerates Hugging Face downloads, and uploads large artifacts to GCS in parallel chunks.

Model Garden custom import

model = model_garden.CustomModel(
    gcs_uri=imported_custom_model_uri,
)
deploy_options = model.list_deploy_options()
print(deploy_options)

Creates a custom Model Garden model from GCS artifacts and checks supported deployment configurations.

Deploy endpoint

endpoint = model.deploy(
    machine_type="g2-standard-24",
    accelerator_type="NVIDIA_L4",
    accelerator_count=2,
)

Registers the model, provisions serving resources, creates a Vertex AI Endpoint, and deploys the model.

Dedicated endpoint prediction

response = endpoint.predict(
    instances=[{"prompt": "how many r does strawberry have?"}],
    use_dedicated_endpoint=True,
)
print(response.predictions)

Runs inference directly through the deployed Vertex AI endpoint.

OpenAI SDK endpoint call

creds, project = google.auth.default()
creds.refresh(google.auth.transport.requests.Request())
endpoint_url = f"https://{endpoint.gca_resource.dedicated_endpoint_dns}/v1beta1/{endpoint.resource_name}"
client = openai.OpenAI(base_url=endpoint_url, api_key=creds.token)
prediction = client.chat.completions.create(model="", messages=[{"role": "user", "content": "Tell me a joke"}], temperature=0.7)

Uses a refreshed Google auth token and dedicated endpoint DNS to call the deployed model through OpenAI-compatible chat completions.

Models & APIs used

  • Models: xsanskarx/thinkygemma-4b
  • APIs / services: Vertex AI, Cloud Storage
  • SDKs / libraries: google-cloud-aiplatform, vertexai, openai, google-auth, google-cloud-storage, huggingface_hub

When to use this

Use this pattern when you need to serve Hugging Face open-model weights on scalable Vertex AI infrastructure through Model Garden custom import.

Gotchas & caveats

  • Vertex AI API must be enabled for the Google Cloud project.
  • A Cloud Storage bucket is required for model artifacts.
  • Hugging Face authentication is needed to download models, especially gated ones.
  • The helper excludes *.bin, *.pth, *.gguf, and .gitattributes by default.
  • Deployment can take 15-20 minutes while hardware is provisioned.
  • The sample deployment uses g2-standard-24 with 2 NVIDIA_L4 GPUs.
  • Delete the endpoint and bucket to avoid ongoing charges.

Best practices

  • Use environment variables as fallbacks for project and region.
  • Enable hf_transfer for faster Hugging Face downloads.
  • Upload large model files to Cloud Storage with parallel chunk transfers.
  • Call list_deploy_options before deployment to verify supported configurations and resource needs.
  • Use a dedicated endpoint for prediction calls.
  • Refresh Google auth credentials before using the OpenAI SDK against the Vertex AI endpoint.
  • Clean up endpoints and buckets after the tutorial.

Graph View

Backlinks