Generative AI Google Brain

gemini-use-cases-document-processing-document-processing

3 min read

Document Processing with Gemini

Source notebook

Repo path: gemini/use-cases/document-processing/document_processing.ipynb · Open on GitHub · intermediate

Processes PDFs with Gemini on Vertex AI for extraction, classification, QA, summarization, translation, and page selection.

Summary

This notebook teaches how to use the Gemini API in Vertex AI through the Google Gen AI SDK for Python to process PDF documents. It demonstrates an end-to-end workflow that installs dependencies, authenticates, creates a GenAI client, sends PDFs from bytes, GCS URIs, and HTTPS URLs, and applies Gemini to structured extraction, classification, QA, summaries, table parsing, translation, comparison, and page extraction.

Key code patterns

Create Vertex AI GenAI client

from google import genai
PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")
LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "us-central1")
client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)

Initializes the Google Gen AI SDK client for Gemini on Vertex AI.

Send local PDF bytes

with open("invoice.pdf", "rb") as f:
    file_bytes = f.read()
response = client.models.generate_content(
    model=MODEL_ID,
    contents=["The following document is an invoice.", Part.from_bytes(data=file_bytes, mime_type=PDF_MIME_TYPE)],
)

Shows how to pass a local PDF directly to Gemini as bytes.

Structured extraction with Pydantic

response = client.models.generate_content(
    model=MODEL_ID,
    contents=[prompt, Part.from_uri(file_uri=gcs_uri, mime_type=PDF_MIME_TYPE)],
    config=GenerateContentConfig(response_schema=Invoice, response_mime_type="application/json"),
)
invoice_data = response.parsed

Constrains Gemini output to a typed schema and reads parsed results.

Enum document classification

response = client.models.generate_content(
    model=MODEL_ID,
    contents=["Classify the following document.", Part.from_uri(file_uri=url, mime_type=PDF_MIME_TYPE)],
    config=GenerateContentConfig(response_schema=DocumentCategory, response_mime_type="text/x.enum"),
)

Limits classification output to predefined document categories.

Classify then extract

classification_response = client.models.generate_content(...)
extraction_schema = classification_to_schema.get(classification_response.parsed)
if extraction_schema:
    extraction_response = client.models.generate_content(
        model=MODEL_ID,
        config=GenerateContentConfig(response_schema=extraction_schema, response_mime_type=JSON_MIME_TYPE),
    )

Routes each document to the appropriate extraction schema based on classification.

Extract relevant PDF pages

response = client.models.generate_content(
    model=MODEL_ID,
    contents=["<Document>", Part.from_uri(file_uri=pdf_path, mime_type=PDF_MIME_TYPE), "</Document>", PROMPT_PAGES.format(question=question)],
    config=GenerateContentConfig(response_mime_type=JSON_MIME_TYPE, response_schema=list[int]),
)
pages = response.parsed

Uses Gemini to identify pages relevant to a question before slicing the PDF.

Models & APIs used

  • Models: gemini-3.5-flash
  • APIs / services: Vertex AI, Cloud Storage
  • SDKs / libraries: google-genai, pypdf, pydantic

When to use this

Use this pattern when building PDF document-processing workflows that need Gemini-based extraction, classification, QA, summarization, translation, comparison, or page filtering.

Gotchas & caveats

  • Colab requires google.colab auth.authenticate_user, while Vertex AI Workbench does not.
  • A Google Cloud project is required and the Vertex AI API must be enabled.
  • The notebook uses billable Vertex AI components.
  • LOCATION defaults to us-central1 from GOOGLE_CLOUD_REGION when unset.
  • When comparing multiple documents, the notebook notes that document order can matter and should be specified in the prompt.
  • For page extraction, the notebook expands selected pages to include each successor page before slicing.

Best practices

  • Use explicit system instructions for extraction, classification, QA, and summarization tasks.
  • Use response_schema and response_mime_type to enforce structured JSON or enum outputs.
  • Use Pydantic models to define extraction schemas with field descriptions.
  • Use Part.from_uri for GCS or HTTPS PDFs and Part.from_bytes for local PDF bytes.
  • Use response.parsed when consuming structured model outputs.
  • Tell the model to use only the provided document as context for page-selection tasks.

Graph View

Backlinks