Generative AI Google Brain

gemini-use-cases-spatial-understanding-object-detection-and-editing

3 min read

Detecting and Editing Visual Objects with Gemini

Source notebook

Repo path: gemini/use-cases/spatial-understanding/object_detection_and_editing.ipynb · Open on GitHub · intermediate

Detects visual objects with Gemini, crops them, and edits them with Gemini image models.

Summary

The notebook teaches open-vocabulary object detection on book photos and other images using Gemini spatial understanding and structured JSON outputs. It builds a workflow that loads images, prompts Gemini for bounding boxes, captions, and labels, denormalizes boxes, crops detected objects, and then restores or transforms the crops with Gemini image editing models. It also shows prompt changes for detecting electronic components without changing the response schema.

Key code patterns

GenAI client setup

from google import genai
 
check_environment()
client = genai.Client()
check_configuration(client)

Uses environment-driven configuration for either Agent Platform or Google AI Studio.

Structured detection schema

class DetectedObject(pydantic.BaseModel):
    box_2d: list[int]
    caption: str
    label: str
 
DetectedObjects = list[DetectedObject]

Matches the prompt output so the SDK can parse detected objects into typed instances.

JSON response config

config = GenerateContentConfig(
    response_mime_type="application/json",
    response_schema=DetectedObjects,
)

Requests structured JSON detection results instead of free-form text.

Generate content with retry

response = client.models.generate_content(
    model=model.value,
    contents=contents,
    config=config,
)

Centralizes Gemini calls and wraps them with retry behavior.

Denormalize boxes

w, h = self.source_image.size
for obj in self.detected_objects:
    y1, x1, y2, x2 = obj.box_2d
    x1, x2 = to_image_coord(x1, w), to_image_coord(x2, w)
    y1, y2 = to_image_coord(y1, h), to_image_coord(y2, h)
    obj.box_2d = [x1, y1, x2, y2]

Converts Gemini 0-1000 [y1,x1,y2,x2] boxes into Pillow pixel coordinates.

Reusable editing step

restore_objects = object_editing_function(
    RESTORATION_PROMPT,
    WorkflowStep.CROPPED,
    WorkflowStep.RESTORED,
    default_config=RESTORATION_CONFIG,
)

Registers image-editing functions as workflow transitions from cropped objects to edited outputs.

Models & APIs used

When to use this

Use this pattern when you need prompt-driven detection, extraction, and cleanup of visual objects without training a custom vision model.

Gotchas & caveats

  • Agent Platform requires a Google Cloud project and the Agent Platform API enabled.
  • Google AI Studio requires a Gemini API key.
  • Preview models require location set to global when using Vertex AI.
  • Gemini bounding boxes are normalized to a 0-1000 coordinate space and use [y1, x1, y2, x2].
  • Ultra-high media resolution can help small visual details but increases tokenized detail.
  • The notebook notes temporary workshop projects may have 1 QPM quota and retry on 429.

Best practices

  • Store environment configuration outside source code with environment variables, Colab Secrets, or platform defaults.
  • Use structured outputs with response_mime_type and response_schema for automated parsing.
  • Keep bounding box granularity controlled through prompt wording.
  • Use precise prompts such as excluding captions from boxes or preserving original line breaks.
  • Use low randomness, seed, and image response modalities for more deterministic restoration outputs.
  • Switch object categories by changing the prompt while keeping the same schema and workflow.

Graph View

Backlinks