Open & Partner Models

Open & Partner Models on Google Cloud is the area covering how engineers use non-Gemini foundation models, open-source models, and partner models through Google Cloud services such as Vertex AI, BigQuery, Cloud Run, GKE, and Agent Engine. It includes deployment, serving, evaluation, fine-tuning, agent integration, and SQL/data workflows for models from Model Garden, Hugging Face, MaaS providers, and partners such as Anthropic.

Synthesized across 34 notebooks. See the source pages under Notebooks below for line-level grounding.

This area is centered on bringing open and partner models into managed Google Cloud workflows without giving up production controls such as IAM, private endpoints, quota-aware scaling, Cloud Storage staging, Model Registry, and cleanup of billable resources.

The notebooks repeatedly show three dominant paths: deploy models from Model Garden or Hugging Face to Vertex AI endpoints, serve models with runtime stacks such as vLLM, TGI, Ollama, or PyTorch DLCs, and call those models from higher-level systems such as BigQuery, ADK agents, LangGraph, LiteLLM, the OpenAI SDK, or Gemini Enterprise.

Evaluation is a first-class workflow: use the Vertex AI Gen AI Evaluation SDK with fixed datasets, reusable metrics, pairwise or pointwise rubrics, experiments, and autoraters before treating a model migration, fine-tune, prompt change, or third-party comparison as meaningful.

Canonical workflow

1. Choose the model family and serving path: Model Garden MaaS, imported Hugging Face weights, custom Vertex AI endpoint, Cloud Run GPU, GKE with vLLM, BigQuery remote model, or partner API on Vertex AI.
2. Enable required APIs, set PROJECT_ID and region, configure authentication, create or select Cloud Storage buckets, and grant only the required IAM roles to service accounts and BigQuery connections.
3. Check model terms, gated-model access, EULAs, regional availability, machine and accelerator compatibility, and GPU or service quotas before deployment.
4. Deploy or import the model with the selected runtime, explicitly configuring predict and health routes, runtime environment variables, scaling settings, machine type, accelerators, and staging artifacts.
5. Smoke test the endpoint or service with a minimal prediction before wiring it into BigQuery, LiteLLM, OpenAI SDK, ADK, LangGraph, Gradio, Cloud Functions, or Gemini Enterprise.
6. Evaluate outputs with fixed datasets, appropriate metrics, rubrics, experiments, autoraters, or baseline comparisons before promoting a model, prompt, adapter, or migration candidate.
7. Tune cost and performance with concurrency, autoscaling, token limits, replica counts, batching, GPU utilization, and model keep-alive settings.
8. Persist useful generated outputs and clean up endpoints, models, experiments, buckets, BigQuery datasets, connections, Cloud Run services, GKE resources, and Agent Engine resources after the workflow.

Best practices

See Open & Partner Models - Best Practices for the full distilled guide.

• Use environment variables or secure storage for project, region, provider API keys, Hugging Face tokens, and other sensitive configuration instead of hardcoding them.
• Check deployment options, region support, EULAs, gated-model license acceptance, IAM roles, and GPU or service quotas before starting long-running deployments.
• Smoke test every deployed endpoint or serving container with a minimal request before connecting it to agents, evaluation jobs, BigQuery, LiteLLM, or application code.
• Use Cloud Storage staging buckets for model artifacts, tuning datasets, evaluation outputs, and Vertex AI initialization, and keep buckets regional when the workflow requires region alignment.
• Use the same dataset, prompt template, baseline, and metrics when comparing model architectures, migration candidates, fine-tuned models, adapters, or prompt variants.
• Track evaluations with Vertex AI Experiments and inspect EvaluationDataset, EvaluationResult, explanations, score distributions, confusion matrices, and item-level agreement before trusting aggregate metrics.

Key models & APIs

• Vertex AI API, Vertex AI Model Garden SDK, Vertex AI Gen AI Evaluation SDK, Vertex AI Model Registry, Vertex AI Agent Engine and reasoning_engines.AdkApp, BigQuery AI.GENERATE, AI.GENERATE_TABLE, ML.GENERATE_EMBEDDING, and remote models, BigQuery Connection API and Cloud resource connections, Cloud Run GPU with Ollama and vLLM, Google Kubernetes Engine with vLLM and Gen2 Cloud Functions, Hugging Face Deep Learning Containers: TGI and PyTorch inference, LiteLLM OpenAI-compatible APIs, OpenAI SDK against Vertex AI MaaS and endpoints, AnthropicVertex client for Claude on Vertex AI, ADK, FunctionTool, InMemorySessionService, and adk.Runner, LangGraph and LangChain RAG workflows, gemini-2.5-flash, gemini-2.5-pro, gemini-2.0-flash, gemini-2.0-flash-lite, gemini-embedding-001, text-embedding-005, Gemma models including google/gemma-2b, google/gemma-7b-it, google/gemma-2-9b-it, google/gemma-2-2b-it, google/gemma-3-4b-it, gemma2, and gemma3, Llama models including meta/llama3_1@llama-3.1-8b-instruct, Llama-3.3-70B-Instruct, Llama 3.2 variants, Llama 4 Scout, and Llama MaaS models, Qwen models including publishers/qwen/models/qwen3@qwen3-1.7b, qwen3:8b, Qwen/Qwen2.5-0.5B, and Qwen 3 BYOD evaluation responses, DeepSeek models including deepseek-ai/deepseek-r1-0528-maas, deepseek-ai/deepseek-v3.1-maas, and deepseek-ai/DeepSeek-R1-Distill-Qwen-7B, Claude models including claude-3-haiku@20240307, claude-3-5-sonnet@20240620, and claude-3-5-sonnet-v2@20241022, OpenAI and GPT-OSS models mentioned in evaluation or MaaS workflows including gpt-5-mini, gpt-4o, unsloth/gpt-oss-20b, and openai/gpt-oss-20b-maas, Hugging Face and partner models including google/paligemma-3b-mix-448, CYFRAGOVPL/PLLuM-12B-chat, black-forest-labs/FLUX.1-dev, AtlaAI/Selene-1-Mini-Llama-3.1-8B, publishers/intfloat/models/e5@multilingual-e5-small, and TimesFM

Notebooks (34)

Embeddings & Vector Search (1)

• Use Gemini and OSS Text-Embedding Models Against Your BigQuery Data — Generate BigQuery embeddings with Gemini and a deployed OSS E5 model.

Gemini (4)

• Use Gen AI Evaluation SDK to Evaluate Models in Vertex AI Studio, Model Garden, and Model Registry — Evaluates Gemini, Llama MaaS, Claude, and prompt templates with Vertex AI Gen AI Evaluation SDK.
• Evaluating Third-Party LLMs with the Vertex AI Gen AI Evaluation SDK — Evaluates third-party, MaaS, BYOM, and Gemini models with Vertex AI Gen AI Evaluation.
• Migrating Foundation Models: A Practical Guide with Gen AI Evaluation Serivce — Compares model migration candidates with Vertex AI Gen AI Evaluation and prompt optimization workflows.
• Introduction to Generative AI functions in BigQuery — Introduces BigQuery generative AI functions for SQL-based text analysis and forecasting.

Open Models (27)

• Building an ADK agent using QWEN 3 on Vertex AI — Builds and deploys an ADK weather agent using Qwen3 on Vertex AI and Agent Engine.
• Accelerate LLM Inference with EAGLE Speculative Decoding on Vertex AI — Benchmarks EAGLE speculative decoding for Llama 4 Scout on Vertex AI against a baseline endpoint.
• Qwen 3 evaluation - Bring your own data eval — Compares fine-tuned and base Qwen 3 medical summaries using Vertex AI pairwise evaluation.
• Using open autorater for running evaluations with Vertex AI Gen AI Evaluation — Deploys Selene as an open judge on Vertex AI and uses Gen AI Evaluation to score LLM responses.
• Hugging Face DLCs: Using Gemma for running evaluations with Vertex AI Gen AI Evaluation — Deploys Gemma 2 on Vertex AI TGI and evaluates summarization with Gen AI Evaluation.
• MetaMath with Vertex AI Open Source Model Tuning — Fine-tunes a Llama 3.1 8B model on MetaMathQA using Vertex AI managed tuning.
• Fine-tuning GPT-OSS 20B with Unsloth on Vertex AI Colab Enterprise and Nvidia A100 40GB GPU — Fine-tunes GPT-OSS 20B with Unsloth LoRA on Vertex AI Colab Enterprise using an A100 GPU.
• Hugging Face DLCs: Fine-tuning Gemma with Transformer Reinforcement Learning (TRL) on Vertex AI — Fine-tunes google/gemma-2b with TRL SFT and LoRA in a Vertex AI custom container job.
• Get started with Vertex AI Model Garden SDK — Deploy and test open models on Vertex AI with the Model Garden SDK.
• Import, Deploy, and Serve custom open models on Vertex AI using Vertex AI Model Garden SDK. — Imports Hugging Face open-model weights to GCS, deploys them on Vertex AI, and serves predictions.
• Get started with Model Garden Terraform Deployment — Deploys Model Garden and Hugging Face open models to Vertex AI endpoints with Terraform.
• Handling Reasoning with MaaS Models on Vertex AI using vLLM — Shows how to call reasoning MaaS models on Vertex AI through the OpenAI SDK.
• Serving Gemma 3 with Ollama on Cloud Run — Deploys Gemma 3 with Ollama as a private GPU-backed Cloud Run API and tests it with curl and Python.
• Running Qwen 3 with Ollama in Cloud Run for Agents — Deploys Qwen3:8b with Ollama on GPU Cloud Run and uses it as an ADK agent model.
• Serving Gemma 3 with vLLM on Cloud Run — Deploys Gemma 3 on Cloud Run with vLLM as an OpenAI-compatible chat API.
• Deploying Multiple LoRA Adapters on Vertex AI with vLLM — Deploys Gemma 2 with multiple LoRA adapters on Vertex AI using a custom vLLM container.
• Running a Gemma 2-based agentic RAG with Ollama on Vertex AI and LangGraph — Deploys a Gemma 2 Ollama container on Vertex AI and uses it in a LangGraph SQL RAG agent.
• Hugging Face DLCs: Serving PaliGemma using Pytorch Inference on Vertex AI with Custom Handler — Deploys gated PaliGemma from Hugging Face to Vertex AI using a PyTorch DLC and custom handler.
• Hugging Face DLCs: Serving PLLuM using Pytorch Inference on Vertex AI with Custom Handler — Deploys CYFRAGOVPL/PLLuM-12B-chat to Vertex AI with a Hugging Face PyTorch custom handler.
• Hugging Face DLCs: Serving Gemma with Text Generation Inference (TGI) on Vertex AI — Deploys gated Gemma 7B IT from Hugging Face to Vertex AI using a TGI deep learning container.
• Hugging Face DLCs: Serving Gemma 2 with multiple LoRA adapters with Text Generation Inference (TGI) on Vertex AI — Deploys Gemma 2 with SQL and code LoRA adapters on Vertex AI using a Hugging Face TGI custom handler.
• Use Any OSS Gen AI Model Against Your BigQuery Data — Deploys Llama 3.3 70B on Vertex AI and calls it from BigQuery ML for medical transcript analytics.
• Cloud Run GPU Inference: Gemma 2 RAG Q&A with Ollama and LangChain — Deploys Gemma 2 on Cloud Run GPU with Ollama and builds a LangChain RAG Q&A chain.
• Deploying Llama 3 on Google Kubernetes Engine with Cloud Functions and vLLM — Deploys Llama 3.2 on GKE with vLLM and exposes it through a Gen2 Cloud Function.
• Guess who or what app using Hugging Face Deep Learning container model on Vertex AI — Builds a Gradio riddle game using Gemini and a Hugging Face FLUX model deployed on Vertex AI.
• Serving Open-Source LLMs on Vertex AI with LiteLLM and OpenAI-Compatible APIs — Deploys a Llama 3.1 Model Garden model on Vertex AI and calls it through LiteLLM OpenAI-style APIs.
• Build and deploy a Hugging Face smolagent using DeepSeek-r1 on Vertex AI — Deploys DeepSeek R1 Distill Qwen 7B on Vertex AI and wraps it in a smolagents math verifier agent.

Partner Models (1)

• Multimodal Function Calling with Claude Models — Uses Claude on Vertex AI for multimodal prompting and tool use with a Wikipedia lookup.

Vertex AI Search (1)

• Open Source Models (Gemma) as a agent with Gemini Enterprise — Deploys Gemma on Cloud Run, wraps it with ADK, deploys to Agent Engine, and registers it in Gemini Enterprise.

Related concepts

• MLOps & Deployment · Evaluation · Applied Use Cases · Agents & ADK · Tuning & Customization · Prompt Engineering · Vision · Agent Engine

---

Part of the Generative AI Google Brain · Open & Partner Models - Best Practices