Generative AI Google Brain

gemini-use-cases-vision-assistant-enhanced-vision-assistant

3 min read

Enhanced Vision Assistant with Gemini

Source notebook

Repo path: gemini/use-cases/vision-assistant/enhanced_vision_assistant.ipynb · Open on GitHub · advanced

Builds a Gemini vision assistant that detects objects, assesses hazards, and speaks navigation guidance.

Summary

This notebook teaches how to combine Google Cloud Vision API, Gemini through the Google Gen AI SDK, and Cloud Text-to-Speech into an accessibility-oriented navigation assistant. The workflow initializes cloud clients, captures camera frames with OpenCV, detects localized objects, estimates depth and hazard priority, tracks navigation context, asks Gemini for scene guidance, and plays prioritized audio feedback.

Key code patterns

Vertex AI Gen AI client

from google import genai
 
PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")
LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "us-central1")
genai_client = genai.Client(
    project=PROJECT_ID,
    location=LOCATION,
    vertexai=True,
)

Configures Gemini access through Vertex AI using project and region settings.

Vision object localization

image = vision_v1.Image(content=content)
features = [
    vision_v1.Feature(type=vision_v1.Feature.Type.OBJECT_LOCALIZATION),
    vision_v1.Feature(type=vision_v1.Feature.Type.LABEL_DETECTION),
]
request = vision_v1.AnnotateImageRequest(image=image, features=features)
response = self.vision_client.annotate_image(request=request)

Uses Cloud Vision API to detect objects and labels from encoded camera frames.

Depth and priority heuristic

height = bbox[2].y - bbox[0].y
width = bbox[2].x - bbox[0].x
if height * width > self.MIN_OBJECT_SIZE:
    depth_estimate = 1 / (height * width)
    priority = self.calculate_priority(obj, bbox, depth_estimate)

Turns bounding-box size into a practical depth estimate and navigation priority.

Safe path sectors

sectors = {
    "left": {"clear": True, "score": 0},
    "center": {"clear": True, "score": 0},
    "right": {"clear": True, "score": 0},
}
for obj in objects:
    center_x = (obj.bbox[0].x + obj.bbox[2].x) / 2

Divides the scene into left, center, and right sectors to recommend navigation direction.

Audio guidance queue

pygame.mixer.init(buffer=512)
self.audio_queue = queue.PriorityQueue()
self.speech_client = texttospeech.TextToSpeechClient()

Combines Text-to-Speech with a priority queue and pygame playback for urgent audio instructions.

Models & APIs used

  • APIs / services: Vertex AI, Cloud Vision API, Cloud Text-to-Speech API
  • SDKs / libraries: google-genai, opencv-python, pygame, numpy, scipy, google-cloud-vision, google-cloud-texttospeech

When to use this

Use this pattern when building a camera-based accessibility assistant that converts detected visual hazards into spoken guidance.

Gotchas & caveats

  • Requires a Google Cloud project with Vertex AI API, Cloud Vision API, and Cloud Text-to-Speech API enabled.
  • Colab authentication is handled only when running in google.colab; other environments need application default credentials or GOOGLE_APPLICATION_CREDENTIALS.
  • A camera and permission to access it are required to run the assistant.
  • Default location is us-central1 unless GOOGLE_CLOUD_REGION is set.
  • Temporary audio MP3 files are created and cleaned up when the assistant stops.

Best practices

  • Use environment variables for GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION when possible.
  • Allow an explicit credentials path but also support application default credentials.
  • Filter small detections with MIN_OBJECT_SIZE before generating guidance.
  • Prioritize navigation guidance by urgency using NavigationPriority.
  • Clean up camera/audio resources and temporary audio files in stop().

Graph View

Backlinks