Generative AI Google Brain

gemini-use-cases-intro-multimodal-use-cases

3 min read

Gemini: An Overview of Multimodal Use Cases

Source notebook

Repo path: gemini/use-cases/intro_multimodal_use_cases.ipynb · Open on GitHub · intermediate

Shows Gemini multimodal prompts for text, PDFs, images, video, audio, code, retail, diagrams, and comparisons.

Summary

After installing google-genai and creating an enterprise Gen AI client for Vertex AI, the notebook calls gemini-2.5-flash with text and Part.from_uri inputs. It demonstrates PDF Q&A and summarization, multi-image reasoning, video description, audio summary/transcription, codebase analysis with context caching, and mixed video/image prompts. It closes with applied examples for retail recommendations, ER diagram interpretation, and image similarity/difference analysis.

Key code patterns

Create Gen AI client

import os
from google import genai
 
PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")
LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "global")
client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)
MODEL_ID = "gemini-2.5-flash"

Configures the Vertex AI-backed Google Gen AI SDK client used for every request.

Send URI media parts

pdf_file = Part.from_uri(
    file_uri="https://arxiv.org/pdf/2507.06261",
    mime_type="application/pdf",
)
response = client.models.generate_content(
    model=MODEL_ID,
    contents=[pdf_file, "How many tokens can the model process?"],
)

Shows the common pattern for passing PDFs, images, video, and audio with explicit MIME types.

Audio transcription config

audio_file = Part.from_uri(file_uri=audio_url, mime_type="audio/mpeg")
config = GenerateContentConfig(max_output_tokens=8192, audio_timestamp=True)
response = client.models.generate_content(
    model=MODEL_ID,
    contents=[audio_file, prompt],
    config=config,
)

Uses output-token control and audio timestamps for structured interview transcription.

Cache large code context

_, code_index, code_text = ingest(repo_url, exclude_patterns=exclude_patterns)
prompt = f"Context:\n{code_index}\n{code_text}"
cached_content = client.caches.create(
    model=MODEL_ID,
    config=CreateCachedContentConfig(contents=prompt, ttl="3600s"),
)
response = client.models.generate_content(
    model=MODEL_ID,
    contents=question,
    config=GenerateContentConfig(cached_content=cached_content.name),
)

Stores the codebase prompt once and reuses it for onboarding, bug finding, and summarization questions.

Label image choices

art_images = [Part.from_uri(file_uri=url, mime_type="image/png") for url in art_image_urls]
contents = [
    "art 1:", art_images[0],
    "art 2:", art_images[1],
    "art 3:", art_images[2],
    "art 4:", art_images[3],
    "room:", room_image,
    prompt,
]

Labels candidate images so the model can rank and reference each option with less hallucination risk.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI, Cloud Storage
  • SDKs / libraries: google-genai, gitingest, nest_asyncio, IPython.display

When to use this

Use this pattern when building Vertex AI Gemini workflows that combine text with PDF, image, video, audio, or code context.

Gotchas & caveats

  • Colab requires google.colab.auth.authenticate_user(); Vertex AI Workbench does not require that cell.
  • An existing Google Cloud project and the Vertex AI API must be enabled before creating the client.
  • The tutorial uses billable Vertex AI components.
  • PROJECT_ID falls back to GOOGLE_CLOUD_PROJECT, and LOCATION falls back to global.
  • Context caching is available only for the models listed in the Context caching documentation.
  • Large repository analysis depends on excluding binary and image files before creating the code prompt.

Best practices

  • Pass non-text inputs with explicit MIME types using Part.from_uri.
  • Use context caching for repeated questions over a large codebase instead of resending the same prompt.
  • Set temperature to 0 for factual identification prompts, as shown for the train-line example.
  • Label candidate images in the prompt when asking the model to choose among provided images.
  • Tell the model not to make up information when answers must be grounded only in attached audio or video.
  • Exclude binary and image assets before ingesting a repository for codebase analysis.

Graph View

Backlinks