Generative AI Google Brain

gemini-use-cases-retrieval-augmented-generation-retail-warranty-claim-chatbot

3 min read

Building a Multimodal Chatbot for Warranty Claims using Gemini and Vector Search in Vertex AI

Source notebook

Repo path: gemini/use-cases/retrieval-augmented-generation/retail_warranty_claim_chatbot.ipynb · Open on GitHub · advanced

Builds a multimodal warranty-claims chatbot with Gemini, RAG, Vector Search, and function calling.

Summary

The notebook teaches how to build a warranty claims chatbot for a fictitious shoe company, AquaStride. It converts PDFs to images, extracts text and tables with Gemini, embeds chunks, loads them into Vertex AI Vector Search, and answers questions from retrieved context. It then configures Gemini prompts and function declarations to analyze shoe-tag and damage images for warranty workflows.

Key code patterns

Initialize Vertex AI models

vertexai.init(project=PROJECT_ID, location=LOCATION)
multimodal_model = GenerativeModel(
    "gemini-2.0-flash",
    generation_config=generation_config,
)
text_embedding_model = TextEmbeddingModel.from_pretrained(
    "text-embedding-005"
)

Sets project, region, Gemini, and embedding model before RAG and image analysis.

Extract PDF page content with Gemini

image = Image.load_from_file(image_path)
response = multimodal_model.generate_content([image, prompt_text])
text_content = response.text
response = multimodal_model.generate_content([image, prompt_table])
table_content = response.text
page_content.append(text_content + "\n" + table_content)

Uses multimodal Gemini calls to turn rendered PDF pages into searchable text.

Chunk and embed documents

loader = DataFrameLoader(df, page_content_column="page_content")
documents = loader.load()
text_splitter = CharacterTextSplitter(
    chunk_size=10000,
    chunk_overlap=200,
)
doc_splits = text_splitter.split_documents(documents)
vector = text_embedding_model.get_embeddings([text])[0].values

Prepares extracted page text for Vector Search by splitting and embedding chunks.

Create Vector Search index

aiplatform.MatchingEngineIndex.create_tree_ah_index(
    display_name=vec_search_index_name,
    contents_delta_uri=bucket_location,
    dimensions=768,
    approximate_neighbors_count=20,
    distance_measure_type="DOT_PRODUCT_DISTANCE",
)

Loads JSONL embeddings from Cloud Storage into a Vertex AI Vector Search index.

Retrieve context and answer

query_embeddings = generate_text_embedding(query)
response = my_index_endpoint.find_neighbors(
    deployed_index_id=DEPLOYED_INDEX_ID,
    queries=[query_embeddings],
    num_neighbors=5,
)
prompt = get_prompt_text(query, context)
result = multimodal_model.generate_content(prompt).text

Combines nearest-neighbor retrieval with Gemini generation constrained to retrieved context.

Declare Gemini functions

fn_json_from_tag = FunctionDeclaration(
    name="extract_json_from_tag",
    description="This function is used to clean JSON packages from text",
    parameters={"type": "object", "properties": {"records": {"type": "array"}}},
)

Defines structured outputs for shoe-tag extraction, shoe damage extraction, and unrelated images.

Models & APIs used

  • Models: gemini-2.0-flash, text-embedding-005
  • APIs / services: Vertex AI, Vector Search, Cloud Storage, Gemini API
  • SDKs / libraries: google-cloud-aiplatform, vertexai, langchain, langchain_google_vertexai, gradio, pymupdf, pillow

When to use this

Use this pattern when building a multimodal customer-support assistant that must search product documents and inspect user-uploaded images.

Gotchas & caveats

  • Colab requires explicit Google Cloud authentication; Vertex AI Workbench does not.
  • The Vertex AI API must be enabled for the selected Google Cloud project.
  • The notebook uses LOCATION set to us-central1.
  • Runtime restart is required after package installation.
  • Embedding and image processing cells include 12-second sleeps to avoid default Vertex AI quota issues.
  • Vector Search index creation can take minutes, and first index endpoint deployment can take around 25 minutes.
  • Deployment IDs cannot contain hyphens, so the notebook replaces ’-’ with ’_‘.
  • The demo uses a public Vector Search endpoint and states access still requires IAM permissions.

Best practices

  • Use a unique lowercase RAG identifier without spaces for generated resources.
  • Keep chunk overlap when splitting retrieved text to preserve context across chunks.
  • Store Vector Search input as JSONL with id and embedding fields.
  • Use retrieved context in the prompt and instruct Gemini to answer only from provided text.
  • Return a fallback response when no matching page source is found.
  • Use a managed database for production workloads to improve performance and efficiency.
  • Use FunctionDeclaration objects to route image-derived intents into structured warranty workflows.
  • Configure safety settings for dangerous content and harassment on the text model.

Graph View

Backlinks