Generative AI Google Brain

open-models-use-cases-model-garden-litellm-inference

3 min read

Serving Open-Source LLMs on Vertex AI with LiteLLM and OpenAI-Compatible APIs

Source notebook

Repo path: open-models/use-cases/model_garden_litellm_inference.ipynb · Open on GitHub · intermediate

Deploys a Llama 3.1 Model Garden model on Vertex AI and calls it through LiteLLM OpenAI-style APIs.

Summary

This notebook shows how to discover deployable open models in Vertex AI Model Garden, deploy a selected Llama model to a Vertex AI endpoint, and configure LiteLLM to route OpenAI-compatible calls to that endpoint. It demonstrates chat completion and OpenAI-style function calling, then deletes the endpoint for cleanup.

Key code patterns

Initialize Vertex AI

PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")
LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "us-central1")
vertexai.init(project=PROJECT_ID, location=LOCATION)

Sets the project and region before using Model Garden and deploying endpoints.

List deployable models

model_garden_models = model_garden.list_deployable_models(
    model_filter="Llama",
    list_hf_models=False,
)
deployable_models = model_garden.list_deployable_models(
    model_filter="llama",
    list_hf_models=True,
)

Discovers Model Garden models and optionally includes Hugging Face gallery models.

Deploy open model

model_id = "meta/llama3_1@llama-3.1-8b-instruct"
model = model_garden.OpenModel(model_id)
endpoint = model.deploy(
    accept_eula=True,
    machine_type="g2-standard-12",
    accelerator_type="NVIDIA_L4",
    accelerator_count=1,
)

Creates a Vertex AI endpoint for the selected open model with explicit hardware settings.

Configure LiteLLM endpoint

os.environ.setdefault("GOOGLE_GENAI_USE_VERTEXAI", "True")
os.environ["VERTEXAI_PROJECT"] = project_id
os.environ["VERTEXAI_LOCATION"] = location_id
os.environ["LITELLM_LOG"] = "DEBUG"
return f"vertex_ai/openai/{endpoint_id}"

Maps a Vertex AI endpoint resource name to LiteLLM’s OpenAI-compatible model format.

Chat completion

response = completion(
    model=deployed_model,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What's the capital of Japan?"},
    ],
)

Uses LiteLLM’s completion API with role-based OpenAI-style chat messages.

Function calling loop

response = litellm.completion(
    model=deployed_model,
    messages=messages,
    tools=tools,
    tool_choice="auto",
)
tool_calls = response.choices[0].message.tool_calls
function_args = json.loads(tool_call.function.arguments)

Shows the standard tool-call flow: declare tools, parse arguments, execute functions, and send tool responses back.

Models & APIs used

  • Models: meta/llama3_1@llama-3.1-8b-instruct
  • APIs / services: Vertex AI, Vertex AI Model Garden
  • SDKs / libraries: google-cloud-aiplatform, vertexai, litellm, openai, google-auth, requests

When to use this

Use this pattern when serving an open-source Model Garden LLM on Vertex AI while keeping an OpenAI-compatible client interface through LiteLLM.

Gotchas & caveats

  • Requires an existing Google Cloud project with Vertex AI API enabled.
  • Colab users must authenticate with google.colab.auth.authenticate_user().
  • Deployment can take several minutes and requires suitable machine and accelerator quota.
  • Models with a EULA require accept_eula=True.
  • The endpoint resource name must match projects/{project}/locations/{location}/endpoints/{endpoint}.
  • Function-call JSON arguments may not always be valid and should be handled with errors.
  • OPENAI_API_KEY is set to a refreshed Google auth token for the LiteLLM call.
  • LITELLM_LOG is set to DEBUG, with a notebook comment questioning production use.

Best practices

  • Check list_deploy_options() before deployment to verify supported configurations and resource needs.
  • Use environment variables for project and region defaults.
  • Validate the endpoint resource name before configuring LiteLLM.
  • Keep backend or external API execution separate from model tool-call argument generation.
  • Append assistant tool calls and tool responses to the message history before the second model call.
  • Delete the endpoint after the notebook run to clean up resources.

Graph View

Backlinks