Generative AI Google Brain

gemini-use-cases-retail-product-attributes-extraction

3 min read

Product attributes extraction and detailed descriptions from images using Gemini 2.0

Source notebook

Repo path: gemini/use-cases/retail/product_attributes_extraction.ipynb · Open on GitHub · intermediate

Extract product attributes and detailed retail descriptions from images with Gemini 2.0 on Vertex AI.

Summary

This notebook teaches how to wrap Gemini 2.0 in a product image agent for retail image understanding. It demonstrates loading images from GCS, HTTP, HTTPS, or local files, generating detailed product descriptions, extracting open- and closed-vocabulary attributes, and using a self-correcting prompt to verify output in one prompt.

Key code patterns

Initialize Vertex AI

PROJECT_ID = "YOUR_PROJECT_ID"
LOCATION = "us-central1"
 
import vertexai
vertexai.init(project=PROJECT_ID, location=LOCATION)

Sets the Google Cloud project and region before creating Gemini models.

Load image as Part

if image_uri.startswith("gs://"):
    return Part.from_uri(image_uri, mime_type=get_mime_from_uri(image_uri))
elif image_uri.startswith(("http://", "https://")):
    response = requests.get(image_uri)
    return Part.from_data(response.content, mime_type=get_mime_from_uri(image_uri))
else:
    image_bytes = open(image_uri, "rb").read()
    return Part.from_data(image_bytes, mime_type=get_mime_from_uri(image_uri))

Converts GCS, web, or local product images into multimodal prompt parts.

Create product image agent

config = GenerationConfig(temperature=temperature, max_output_tokens=max_output_tokens)
self.gemini_model = GenerativeModel(
    gemini_model_version,
    generation_config=config,
    system_instruction=sys_inst,
)

Centralizes generation settings and retail-specific system instructions.

Generate description

prompt = """
Please write a complete and detailed product description for the
above product image. The length of the description should be at least
200 words.
"""
model_response = self.gemini_model.generate_content([image_part, prompt])
return model_response.text

Uses an image plus text prompt to produce a long product description.

Parse JSON response

lines = answer.split("```")
try:
    answer = lines[-2]
    if answer.startswith("json"):
        answer = answer[4:]
    result = json.loads(answer)
except json.JSONDecodeError:
    answer = lines[1]
    result = json.loads(answer)
return json.dumps(result)

Extracts JSON from markdown-formatted model output for downstream use.

Models & APIs used

When to use this

Use this pattern when retail teams need Gemini to describe product images or extract image-grounded attributes with optional vocabularies.

Gotchas & caveats

  • Requires an existing Google Cloud project with the Vertex AI API enabled.
  • The notebook installs google-cloud-aiplatform and requires a runtime restart.
  • Colab users must authenticate with google.colab.auth.authenticate_user().
  • The default region is us-central1.
  • HTTP image loading returns an empty result path when the fetch status is not 200.
  • JSON parsing assumes the model returns fenced markdown containing JSON.
  • Missing image attributes are instructed to return null in the model output.

Best practices

  • Use temperature 0 for deterministic attribute extraction.
  • Provide system instructions that constrain answers to visible product evidence.
  • Use a closed vocabulary when attribute values must come from an approved set.
  • Parse model JSON output before returning it to application code.
  • Use debug mode to display the image and prompt during development.
  • Ask the model to check and verify results with a self-correcting prompt.

Graph View

Backlinks