Generative AI Google Brain

gemini-use-cases-multimodal-sentiment-analysis-intro-to-multimodal-sentiment-analysis

3 min read

Multimodal Sentiment Analysis with Gemini

Source notebook

Repo path: gemini/use-cases/multimodal-sentiment-analysis/intro_to_multimodal_sentiment_analysis.ipynb · Open on GitHub · intro

Compares Gemini sentiment analysis on audio versus a transcript of the same conversation.

Summary

This notebook teaches how to use Gemini with the Google Gen AI SDK on Vertex AI for multimodal sentiment analysis. It loads a WAV file from Cloud Storage, analyzes sentiment directly from audio, generates a transcript, analyzes sentiment from text, then asks Gemini to compare both analyses. The workflow highlights that audio can capture tone, inflection, and other non-verbal cues that text alone may miss.

Key code patterns

Create Vertex AI GenAI client

from google import genai
 
PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")
LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "us-central1")
client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)

Initializes the Google Gen AI SDK client for Vertex AI with project and region.

Load audio from Cloud Storage

from google.genai.types import Part
 
audio_part = Part.from_uri(
    file_uri="gs://github-repo/generative-ai/gemini/use-cases/multimodal-sentiment-analysis/sample_conversation.wav",
    mime_type="audio/wav",
)

Wraps a Cloud Storage WAV file as a multimodal input part for Gemini.

Analyze sentiment from audio

prompt = "Provide a sentiment analysis of this conversation. Use speaker A, speaker B, etc to identify speakers."
response = client.models.generate_content(
    model=MODEL_ID,
    contents=[audio_part, prompt],
)
audio_analysis = response.text

Sends audio and prompt together so Gemini can use acoustic cues directly.

Transcribe audio with Gemini

prompt = "Generate a transcript of this conversation. Use speaker A, speaker B, etc to identify speakers."
response = client.models.generate_content(
    model=MODEL_ID,
    contents=[audio_part, prompt],
)
transcript = response.text

Uses the same model to produce a speaker-labeled transcript for text-only analysis.

Compare audio and text analyses

prompt = "Provide a short comparison of two analyses of an audio conversation." + text_analysis + audio_analysis
response = client.models.generate_content(
    model=MODEL_ID,
    contents=prompt,
    config=GenerateContentConfig(response_modalities=["TEXT"]),
)
comparison = response.text

Asks Gemini to synthesize differences between transcript-based and audio-based sentiment results.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI, Cloud Storage
  • SDKs / libraries: google-genai

When to use this

Use this pattern when sentiment depends on both spoken words and audio delivery cues such as tone or inflection.

Gotchas & caveats

  • Vertex AI API must be enabled for the Google Cloud project.
  • Colab requires explicit Google Cloud authentication.
  • The notebook installs google-genai and restarts the kernel before imports.
  • The audio input is loaded from a Cloud Storage URI with mime_type audio/wav.
  • This tutorial uses billable Vertex AI components.
  • Default location is us-central1 unless GOOGLE_CLOUD_REGION is set.

Best practices

  • Set project and location before creating the Gen AI client.
  • Use Part.from_uri with an explicit MIME type for Cloud Storage audio inputs.
  • Compare direct audio analysis with transcript-based analysis to understand modality differences.
  • Use speaker labels in prompts for conversation analysis.
  • Set response_modalities to TEXT when requesting text-only output.

Graph View

Backlinks