Generative AI Google Brain

gemini-use-cases-spatial-understanding-spatial-understanding

3 min read

Spatial understanding with Gemini 3

Source notebook

Repo path: gemini/use-cases/spatial-understanding/spatial_understanding.ipynb · Open on GitHub · intermediate

Uses Gemini 2.5 Flash on Vertex AI to detect objects and draw normalized bounding boxes on images.

Summary

This notebook teaches object detection and spatial understanding with the Gemini API in Vertex AI. It installs the Google Gen AI SDK, authenticates in Colab, creates an enterprise GenAI client, and calls Gemini 2.5 Flash with image URIs and prompts. It constrains responses to JSON bounding boxes with Pydantic, then uses PIL to scale normalized coordinates and overlay labeled boxes on sample images.

Key code patterns

Create Vertex GenAI client

from google import genai
 
client = genai.Client(
    enterprise=True,
    project=PROJECT_ID,
    location="global",
)

Initializes the Google Gen AI SDK client for Vertex AI calls.

Structured bounding box schema

class BoundingBox(BaseModel):
    box_2d: list[int]
    label: str
 
config = GenerateContentConfig(
    response_mime_type="application/json",
    response_schema=list[BoundingBox],
)

Forces Gemini output into parseable JSON bounding boxes.

Image prompt with Part.from_uri

response = client.models.generate_content(
    model=MODEL_ID,
    contents=[
        prompt,
        Part.from_uri(file_uri=image_uri, mime_type="image/jpeg"),
    ],
    config=config,
)

Combines a text prompt and remote image input for spatial detection.

Scale normalized coordinates

abs_y_min = int(bbox.box_2d[0] / 1000 * height)
abs_x_min = int(bbox.box_2d[1] / 1000 * width)
abs_y_max = int(bbox.box_2d[2] / 1000 * height)
abs_x_max = int(bbox.box_2d[3] / 1000 * width)

Converts Gemini’s normalized 0-1000 box coordinates to image pixels.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI, Gemini API
  • SDKs / libraries: google-genai, pillow, pydantic

When to use this

Use this pattern when you need Gemini to locate, label, search for, or reason about objects and regions in images.

Gotchas & caveats

  • Requires an existing Google Cloud project with the Vertex AI API enabled.
  • Colab requires auth.authenticate_user().
  • PROJECT_ID must be set or available from GOOGLE_CLOUD_PROJECT.
  • Notebook uses LOCATION = “global”.
  • The system instruction limits output to 25 objects and asks for bounding boxes, not masks.
  • The notebook states there is no magical prompt and encourages prompt experimentation.
  • The notebook notes resizing to 1024px can help the model get the bigger picture, but there is no clear rule.

Best practices

  • Use response_mime_type=“application/json” and response_schema=list[BoundingBox] for controlled generation.
  • Put repeated output-format rules in system_instruction to keep prompts shorter.
  • Use safety_settings with HARM_CATEGORY_DANGEROUS_CONTENT set to BLOCK_ONLY_HIGH.
  • Give repeated objects unique labels based on distinct characteristics such as color, size, or position.
  • Scale normalized box coordinates by image width and height before drawing overlays.
  • Experiment with prompts and images because the notebook states there is no magical prompt.

Graph View

Backlinks