Generative AI Google Brain

embeddings-intro-multimodal-embeddings

4 min read

Introduction to Gemini Multimodal Embeddings

Source notebook

Repo path: embeddings/intro_multimodal_embeddings.ipynb · Open on GitHub · intermediate

Generates Gemini multimodal embeddings and applies them to search, similarity, PDFs, and RAG.

Summary

This notebook teaches how to use Gemini Embedding 2 through the Google GenAI SDK to embed text, images, audio, video, and PDFs. It demonstrates batch text embedding, dimensionality truncation, multimodal aggregation, cosine or dot-product similarity, product and video search, PDF similarity analysis, and task-specific retrieval embeddings for a simple RAG flow. It finishes by retrieving the best matching document passage and using Gemini to generate an answer grounded in that passage.

Key code patterns

Create enterprise client

from google import genai
 
client = genai.Client(
    enterprise=True,
    project=PROJECT_ID,
    location="global",
)

Configures the Google GenAI SDK client for Agent Platform with a project and location.

Batch text embeddings

response = client.models.embed_content(
    model="gemini-embedding-2",
    contents=[
        "How do I get a driver's license?",
        "How do I renew my driver's license?",
    ],
)
embeddings = [e.values for e in response.embeddings]

Embeds multiple text prompts in one API call for efficiency.

Truncate embedding dimensionality

response = client.models.embed_content(
    model="gemini-embedding-2",
    contents=["Hello world"],
    config=EmbedContentConfig(output_dimensionality=768),
)

Uses output_dimensionality to reduce vector size from the 3072-dimensional default.

Embed local image bytes

with open("cookies.png", "rb") as f:
    image_bytes = f.read()
 
response = client.models.embed_content(
    model=MODEL_ID,
    contents=[Part.from_bytes(data=image_bytes, mime_type="image/png")],
)

Shows how to send image bytes with an explicit MIME type.

Aggregate text and image

response = client.models.embed_content(
    model=MODEL_ID,
    contents=[Content(parts=[
        Part(text="An image of cookies"),
        Part.from_bytes(data=image_bytes, mime_type="image/png"),
    ])],
)

A single Content object with multiple parts produces one aggregated embedding.

Embed media from URI

response = client.models.embed_content(
    model=MODEL_ID,
    contents=[Part.from_uri(
        file_uri=AUDIO_URL,
        mime_type="audio/wav",
    )],
)

Embeds Cloud Storage-hosted audio or video by URI instead of local bytes.

Truncate and embed PDF

def truncate_pdf_bytes(file_path, max_pages=6):
    with pymupdf.open(file_path) as doc:
        if doc.page_count <= max_pages:
            return doc.tobytes()
        doc.select(range(max_pages))
        return doc.tobytes(garbage=3, deflate=True)
 
pdf_bytes = truncate_pdf_bytes("sample.pdf")

Handles the notebook’s six-page PDF limit before embedding a document.

Vector search with dot product

embeddings_matrix = np.stack(df["image_embeddings"].values)
df["score"] = embeddings_matrix @ query_emb
top_results = df.nlargest(top_n, "score")

Ranks precomputed embeddings against a query embedding using vectorized dot product.

Task-specific RAG embeddings

doc_emb = client.models.embed_content(
    model=MODEL_ID,
    contents=doc,
    config=EmbedContentConfig(task_type="RETRIEVAL_DOCUMENT"),
)
query_emb = client.models.embed_content(
    model=MODEL_ID,
    contents=query,
    config=EmbedContentConfig(task_type="RETRIEVAL_QUERY"),
)

Uses retrieval-specific task types for document and query embeddings in a RAG workflow.

Models & APIs used

  • Models: gemini-embedding-2, gemini-3.5-flash
  • APIs / services: Agent Platform API, Multimodal Embeddings API, Cloud Storage
  • SDKs / libraries: google-genai, numpy, pandas, seaborn, scikit-learn, pymupdf, matplotlib, IPython

When to use this

Use this pattern when building semantic search, multimodal similarity, or RAG over text, images, audio, video, and PDFs with Gemini embeddings.

Gotchas & caveats

  • Google Colab requires google.colab.auth.authenticate_user().
  • A Google Cloud project is required and the Agent Platform API must be enabled.
  • The client is initialized with location set to global.
  • Default embedding size is 3072 dimensions unless output_dimensionality is specified.
  • Text supports up to 8192 tokens.
  • Image requests allow a maximum of 6 PNG or JPEG images.
  • PDF embedding supports a maximum of 6 pages, so longer PDFs must be truncated.
  • Audio is limited to 80 seconds and MP3 or WAV formats.
  • Video is limited to 128 seconds and MP4 or MOV formats.
  • The video example parses embedding strings with eval.
  • Query embeddings must use a dimensionality compatible with the precomputed embeddings being searched.

Best practices

  • Embed multiple text prompts in one API call for efficiency.
  • Use output_dimensionality to reduce storage cost and improve search speed when lower dimensions are acceptable.
  • Choose input structure deliberately: one Content object aggregates parts into one embedding, while multiple contents return separate embeddings.
  • Create post-level multimodal representations by aggregating separate embeddings, for example by averaging.
  • Truncate PDFs before embedding when they exceed the six-page limit.
  • Use vectorized dot product or cosine similarity for efficient ranking and similarity analysis.
  • Use RETRIEVAL_QUERY and RETRIEVAL_DOCUMENT task types for search and RAG systems.
  • In the generation prompt, instruct the model to answer only from the retrieved source text.

Graph View

Backlinks