Generative AI Google Brain

open-models-use-cases-vertex-ai-deepseek-smolagents

4 min read

Build and deploy a Hugging Face smolagent using DeepSeek-r1 on Vertex AI

Source notebook

Repo path: open-models/use-cases/vertex_ai_deepseek_smolagents.ipynb · Open on GitHub · advanced

Deploys DeepSeek R1 Distill Qwen 7B on Vertex AI and wraps it in a smolagents math verifier agent.

Summary

The notebook shows how to upload a Hugging Face DeepSeek model to Vertex AI Model Registry, deploy it to a Vertex AI endpoint, and call it with a vLLM chat-completions payload. It then builds a Hugging Face smolagents CodeAgent that uses Gemini for orchestration and the deployed DeepSeek endpoint as a math verification tool. The workflow evaluates the agent with Vertex AI Gen AI Evaluation metrics and deploys the custom smolagents app to Vertex AI Reasoning Engine.

Key code patterns

Initialize Vertex AI with staging bucket

PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")
LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "us-central1")
BUCKET_URI = f"gs://{PROJECT_ID}-bucket"
! gsutil mb -p $PROJECT_ID -l $LOCATION $BUCKET_URI
vertexai.init(project=PROJECT_ID, location=LOCATION, staging_bucket=BUCKET_URI)

Sets project, region, and Cloud Storage staging before uploading and deploying resources.

Upload Hugging Face model with vLLM container

deepseek_model = aiplatform.Model.upload(
    display_name=MODEL_ID.replace("/", "--").lower(),
    serving_container_image_uri="us-docker.pkg.dev/deeplearning-platform-release/vertex-model-garden/vllm-inference.cu121.0-6.ubuntu2204.py310",
    serving_container_args=["python", "-m", "vllm.entrypoints.api_server", f"--model={MODEL_ID}", "--tensor-parallel-size=1", "--max-model-len=16384", "--enforce-eager"],
    serving_container_environment_variables={"HF_TOKEN": get_token(), "DEPLOY_SOURCE": "notebook"},
)

Registers the Hugging Face DeepSeek model in Vertex AI using the Model Garden vLLM serving image.

Deploy model to GPU endpoint

deepseek_endpoint = aiplatform.Endpoint.create(
    display_name=MODEL_ID.replace("/", "--").lower() + "-endpoint"
)
deployed_deepseek_model = deepseek_model.deploy(
    endpoint=deepseek_endpoint,
    machine_type="g2-standard-12",
    accelerator_type="NVIDIA_L4",
    accelerator_count=1,
    sync=False,
)

Creates an online prediction endpoint backed by an NVIDIA L4 GPU deployment.

Call deployed model with chat-completions payload

prediction_request = {"instances": [{
    "@requestFormat": "chatCompletions",
    "messages": [{"role": "user", "content": "Count the number of 'r' in the word Strawberry"}],
    "max_tokens": 2048,
    "temperature": 0.7,
}]}
output = deployed_deepseek_model.predict(instances=prediction_request["instances"])

Uses Vertex AI endpoint prediction with the vLLM OpenAI-style chat-completions request format.

OpenAI client for Vertex AI endpoint

self.client = openai.OpenAI(
    base_url=f"https://{location}-aiplatform.googleapis.com/v1beta1/projects/{project_id}/locations/{location}/endpoints/{endpoint_id}",
    api_key=self.credentials.token,
)
response = self.client.chat.completions.create(**completion_kwargs)

Adapts smolagents Model to call Vertex AI through the OpenAI-compatible client interface.

DeepSeek endpoint as smolagents tool

class DeepSeekMathVerifierTool(Tool):
    name = "math_verifier"
    inputs = {"content": {"type": "string", "description": "a text containing math"}}
    output_type = "string"
    def forward(self, content: str):
        return self.endpoint.predict(instances=[{"@requestFormat": "chatCompletions", "messages": [{"role": "user", "content": str(content)}]}])

Wraps the deployed DeepSeek endpoint as a callable tool for agent-side math verification.

Evaluate tool trajectories

response_tool_metrics = [
    "trajectory_exact_match",
    "trajectory_in_order_match",
    "coherence",
]
response_eval_tool_task = EvalTask(dataset=eval_data, metrics=response_tool_metrics, experiment=EXPERIMENT_NAME)
response_eval_tool_result = response_eval_tool_task.evaluate(experiment_run_name=EXPERIMENT_RUN_NAME, runnable=agent_parsed_response)

Runs Vertex AI Gen AI Evaluation on agent responses and tool-use trajectories.

Deploy custom agent to Reasoning Engine

remote_custom_agent = reasoning_engines.ReasoningEngine.create(
    local_custom_agent,
    requirements=["google-cloud-aiplatform[reasoningengine]", "openai", "smolagents", "cloudpickle==3.0.0", "pydantic>=2.10", "requests"],
)
output = remote_custom_agent.query(input="Count the number of 'r' in the word Strawberry. Verify the answer")

Packages and deploys the smolagents application as a managed Vertex AI Reasoning Engine app.

Models & APIs used

  • Models: deepseek-ai/DeepSeek-R1-Distill-Qwen-7B, google/gemini-2.0-flash
  • APIs / services: Vertex AI, Vertex AI Model Registry, Vertex AI Prediction, Vertex AI Reasoning Engine, Vertex AI Gen AI Evaluation, Vertex AI Experiments, Cloud Storage
  • SDKs / libraries: google-cloud-aiplatform, vertexai, openai, smolagents, huggingface_hub, google-auth, pandas, plotly

When to use this

Use this pattern when you need a managed Vertex AI agent that orchestrates with Gemini and calls a Hugging Face open model endpoint as a custom tool.

Gotchas & caveats

  • Vertex AI API must be enabled in an existing Google Cloud project.
  • Colab users must authenticate with google.colab.auth.authenticate_user().
  • A Hugging Face token is required and passed as HF_TOKEN for downloading the model from Hugging Face Hub.
  • The notebook creates a Cloud Storage bucket with gsutil before Vertex AI initialization.
  • Deployment uses NVIDIA_L4 GPU resources and notes that enough GPU quota is required.
  • The notebook states model deployment can take around 20 minutes.
  • Runtime restart is required after installing packages.
  • The custom OpenAI client uses refreshed Google Cloud credentials as the API key and refreshes tokens in a background thread.
  • Cleanup flags default to False, so bucket, endpoint, model, and remote agent are not deleted unless changed.

Best practices

  • Use environment variables GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION as fallbacks for project and location.
  • Initialize vertexai with project, location, and staging_bucket before creating Vertex AI resources.
  • Use Vertex AI Model Registry to manage the imported Hugging Face model lifecycle.
  • Use a dedicated endpoint display name derived from the model ID.
  • Set explicit serving container predict route, health route, port, and environment variables for the vLLM container.
  • Define smolagents tools with name, description, inputs, and output_type.
  • Parse agent logs into predicted_trajectory before sending results to evaluation.
  • Evaluate both response quality and tool trajectory with trajectory_exact_match, trajectory_in_order_match, and coherence.
  • Specify Reasoning Engine deployment requirements explicitly.

Graph View

Backlinks