Generative AI Google Brain

gemini-multimodal-live-api-real-time-rag-bank-loans-gemini-2-0

4 min read

Interactive Loan Application Assistant (Financial Services)

Source notebook

Repo path: gemini/multimodal-live-api/real_time_rag_bank_loans_gemini_2_0.ipynb · Open on GitHub · advanced

Builds a Gemini 2.0 loan document assistant with RAG, large context, audio, Vertex AI Search, and Vector Search.

Summary

This notebook teaches how to answer questions over loan PDFs using Gemini 2.0 through the Google Gen AI SDK on Vertex AI. It demonstrates setup, PDF extraction, custom chunk-and-embed RAG with text-embedding-005, text and audio responses through the Multimodal Live API, large-context prompting, Vertex AI Search grounding, and Vertex AI Vector Search backed RAG.

Key code patterns

Initialize GenAI on Vertex AI

client = genai.Client(
    vertexai=True,
    project=PROJECT_ID,
    location=LOCATION,
)

Routes Google Gen AI SDK calls through Vertex AI for the selected project and region.

Text generation with response modality

response = client.models.generate_content(
    model=MODEL,
    contents=prompt,
    config=GenerateContentConfig(
        response_modalities=["TEXT"],
    ),
)
return response.text

Shows text-in, text-out generation with Gemini 2.0 and explicit response modality.

Live API audio response

config = LiveConnectConfig(response_modalities=["AUDIO"])
async with client.aio.live.connect(model=MODEL_ID, config=config) as session:
    await session.send(input=prompt, end_of_turn=True)
    async for message in session.receive():
        if message.server_content.model_turn:
            ...

Uses the Multimodal Live API to stream generated audio from a text prompt.

Embed document chunks

response = embedding_client.models.embed_content(
    model=embedding_model,
    contents=[text],
    config=EmbedContentConfig(output_dimensionality=768),
)
return [response.embeddings[0].values]

Creates 768-dimensional embeddings for PDF chunks using text-embedding-005.

Retrieve top chunks by cosine similarity

query_embedding = get_embeddings(client, text_embedding_model, query)
similarities = [
    cosine_similarity(query_embedding, chunk_emb)[0][0]
    for chunk_emb in vector_db["embeddings"]
]
top_indices = np.argsort(similarities)[-top_k:]

Implements a simple local vector retrieval step before asking Gemini to answer.

Vertex AI Search tool grounding

vertex_ai_search_tool = Tool(
    retrieval=Retrieval(
        vertex_ai_search=VertexAISearch(datastore=datastore_path)
    )
)
response = client.models.generate_content(
    model=MODEL_ID,
    contents=query,
    config=GenerateContentConfig(tools=[vertex_ai_search_tool], response_modalities=["TEXT"]),
)

Lets Gemini retrieve from a pre-created Vertex AI Search datastore.

Vertex Vector Search RAG corpus

vector_db = rag.VertexVectorSearch(
    index=my_index.resource_name,
    index_endpoint=my_index_endpoint.resource_name,
)
rag_corpus = rag.create_corpus(display_name=DISPLAY_NAME, vector_db=vector_db)

Connects Vertex AI RAG Engine to a Vector Search index and endpoint.

Models & APIs used

  • Models: gemini-2.0-flash-live-preview-04-09, text-embedding-005
  • APIs / services: Vertex AI, Cloud Storage, Multimodal Live API, Vertex AI Search, Vertex AI Vector Search
  • SDKs / libraries: google-genai, google-cloud-storage, gcsfs, PyPDF2, vertexai, pandas, numpy, scikit-learn, tenacity

When to use this

Use this pattern when building a financial document Q&A assistant that needs grounded answers, optional audio output, and scalable retrieval options on Vertex AI.

Gotchas & caveats

  • Gemini 2.0 Flash live preview and google-genai 0.1.0 are described as experimental with variable output.
  • A Google Cloud project and enabled Vertex AI API are required.
  • Colab authentication is needed when running in Google Colab.
  • LOCATION defaults to us-central1 when GOOGLE_CLOUD_REGION is not set.
  • API_ENDPOINT is read from the environment but must be set by the user for helper usage.
  • Embedding and answer generation include retry logic and RESOURCE_EXHAUSTED handling for quota issues.
  • Vertex AI Search assumes an existing datastore with documents already created.
  • Vertex AI Vector Search is paid and requires index settings such as STREAM_UPDATE, 768 dimensions, and a compatible distance measure.
  • Deploying a Vector Search index endpoint for the first time can take about 30 minutes.
  • The VertexRagStore comment states only one corpus is currently allowed.

Best practices

  • Ground generated answers in retrieved document chunks instead of asking the model without context.
  • Include page number and chunk number in citations when generating RAG answers.
  • Use retry with exponential backoff around embedding and generation calls for quota management.
  • Skip unreadable, empty, or blank PDF pages during extraction.
  • Use chunking and embeddings for faster targeted retrieval over large loan documents.
  • Use large-context prompting when questions require information spread across the whole document.
  • Use managed retrieval services such as Vertex AI Search or Vertex AI Vector Search when scalability and infrastructure management matter.

Graph View

Backlinks