Generative AI Google Brain

gemini-use-cases-retail-multimodal-retail-recommendations

3 min read

Multimodal retail recommendation: using Gemini to recommend items based on images and image reasoning

Source notebook

Repo path: gemini/use-cases/retail/multimodal_retail_recommendations.ipynb · Open on GitHub · intro

Builds a Gemini multimodal retail recommender that explains chair fit for a room image.

Summary

This notebook teaches how to use Gemini through Vertex AI with text and image prompts for retail recommendations. It starts by describing a room image, then asks for open-ended furniture and chair recommendations, and finally provides four chair images for grounded comparison. The workflow also demonstrates streaming responses and JSON-formatted output for integrating recommendations into an application.

Key code patterns

Initialize GenAI on Vertex AI

from google import genai
 
PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")
LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "us-central1")
client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)

Configures the Google GenAI SDK to call Gemini through Vertex AI in a specific project and region.

Create image parts from URIs

from google.genai.types import Part
 
room_image = Part.from_uri(
    file_uri=room_image_url,
    mime_type="image/jpeg",
)

Represents hosted JPEG images as multimodal input parts for Gemini prompts.

Mix images and text in one prompt

contents = [
    room_image,
    "Describe what's visible in this room and the overall atmosphere:",
]
responses = client.models.generate_content_stream(
    model=MODEL_ID,
    contents=contents,
)

Shows the core multimodal pattern: combine image and text in ordered contents for visual understanding.

Label candidate images

contents = [
    "Consider the following chairs:",
    "chair 1:", furniture_images[0],
    "chair 2:", furniture_images[1],
    "room:", room_image,
    "For each chair, explain whether it would be appropriate",
]

Labels image options so the model can reference each candidate and reduce hallucinations.

Request JSON output

responses = client.models.generate_content_stream(
    model=MODEL_ID,
    contents=contents,
    config=GenerateContentConfig(response_mime_type="application/json"),
)

Constrains the response format for easier integration with recommendation systems.

Models & APIs used

  • Models: gemini-3.5-flash
  • APIs / services: Vertex AI, Cloud Storage
  • SDKs / libraries: google-genai

When to use this

Use this pattern when you need a quick multimodal retail recommender that compares provided product images against a customer scene and explains fit.

Gotchas & caveats

  • Uses billable Vertex AI components.
  • Colab requires an explicit Google Cloud authentication step.
  • PROJECT_ID must be provided or available as GOOGLE_CLOUD_PROJECT.
  • LOCATION defaults to us-central1 from GOOGLE_CLOUD_REGION when unset.
  • Image inputs are loaded from URI with mime_type=“image/jpeg”.
  • The notebook notes that labeling item numbers in prompts helps reduce hallucinations and improve results.

Best practices

  • Combine text and image in a single prompt for visual understanding.
  • Label each candidate image within the prompt so the model can reference options clearly.
  • Provide the available item images when recommendations should be limited to store inventory.
  • Use response_mime_type=“application/json” when recommendations need to plug into an application.

Graph View

Backlinks