Generative AI Google Brain

gemini-use-cases-document-processing-sheet-music

3 min read

Sheet Music Analysis with Gemini

Source notebook

Repo path: gemini/use-cases/document-processing/sheet_music.ipynb · Open on GitHub · intermediate

Uses Gemini on Vertex AI to extract and edit sheet music PDF metadata and identify a song from audio.

Summary

This notebook teaches how to use the Google Gen AI SDK with Vertex AI to send multimodal prompts containing sheet music PDFs and audio. It extracts structured metadata from a music book, identifies a performed song by comparing audio to a PDF, then asks for JSON metadata and writes it back into a PDF with pypdf.

Key code patterns

Create Vertex AI GenAI client

from google import genai
 
PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")
LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "global")
client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)

Configures the Google Gen AI SDK to call Gemini through Vertex AI.

Prompt Gemini with a PDF from GCS

response = client.models.generate_content(
    model=MODEL_ID,
    contents=[
        sheet_music_extraction_prompt,
        Part.from_uri(file_uri=sheet_music_pdf_uri, mime_type="application/pdf"),
    ],
    config=config,
)

Shows direct multimodal PDF input from Cloud Storage without local file loading.

Compare PDF and audio inputs

pdf_part = Part.from_uri(file_uri=sheet_music_pdf_uri, mime_type="application/pdf")
audio_part = Part.from_uri(file_uri="gs://github-repo/use-cases/sheet-music/24ItalianClip.mp3", mime_type="audio/mpeg")
response = client.models.generate_content(
    model=MODEL_ID,
    contents=[pdf_part, audio_part, song_identification_prompt],
    config=config,
)

Demonstrates cross-modal reasoning over sheet music and an audio performance.

Request JSON and update PDF metadata

config.response_mime_type = "application/json"
response = client.models.generate_content(
    model=MODEL_ID,
    contents=[sheet_music_extraction_prompt, file_part],
    config=config,
)
new_metadata = json.loads(response.text)
writer.add_metadata(new_metadata)

Uses JSON output from Gemini as structured input for PDF metadata editing.

Models & APIs used

  • Models: gemini-3.5-flash
  • APIs / services: Vertex AI, Cloud Storage
  • SDKs / libraries: google-genai, pypdf

When to use this

Use this pattern when extracting or enriching metadata from music documents and combining PDF and audio evidence with Gemini.

Gotchas & caveats

  • Requires an existing Google Cloud project with the Vertex AI API enabled.
  • Colab requires explicit google.colab authentication; Vertex AI Workbench does not.
  • The notebook defaults LOCATION to global from GOOGLE_CLOUD_REGION when unset.
  • Input files are loaded from gs:// URIs, so Cloud Storage access is required.
  • The PDF metadata update downloads the file locally with gcloud storage cp before using pypdf.
  • JSON parsing assumes Gemini returns valid JSON after response_mime_type is set.

Best practices

  • Use environment variables for GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION when notebook parameters are not provided.
  • Set safety settings and a domain-specific system instruction before generation.
  • Use Part.from_uri with explicit MIME types for PDF and audio inputs.
  • Use response_mime_type=“application/json” before parsing model output with json.loads.
  • Copy PDF pages into a writer before adding metadata and saving the file.

Graph View

Backlinks