Generative AI Google Brain

gemini-use-cases-document-processing-tax-automation

2 min read

Automating Income Taxes with Gemini

Source notebook

Repo path: gemini/use-cases/document-processing/tax_automation.ipynb · Open on GitHub · intermediate

Classifies tax PDFs and extracts structured tax fields with Gemini on Vertex AI.

Summary

This notebook demonstrates a document understanding workflow for sample W-2, 1099-DIV, and 1099-INT tax PDFs. It uses the Google Gen AI SDK with Vertex AI to classify each PDF into an enum-backed document type, then extracts fields using document-specific Pydantic response schemas. The extracted data is normalized into a pandas DataFrame and saved as tax_data.csv for further processing.

Key code patterns

Vertex GenAI client

from google import genai
 
PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")
LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "us-central1")
client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)

Creates a Vertex AI-backed Google Gen AI client using project and region settings.

Enum classification

response = client.models.generate_content(
    model=MODEL_ID,
    contents=["Classify the following document.", Part.from_uri(file_uri, PDF_MIME_TYPE)],
    config=GenerateContentConfig(
        temperature=0,
        response_schema=DocumentType,
        response_mime_type="text/x.enum",
    ),
)
return response.parsed

Constrains classification output to predefined tax document types.

Schema-based extraction

response = client.models.generate_content(
    model=MODEL_ID,
    contents=[f"Extract from the following {row['classification'].value} document.", Part.from_uri(row["file_uri"], PDF_MIME_TYPE)],
    config=GenerateContentConfig(
        temperature=0,
        response_schema=document_mapping.get(row["classification"]),
        response_mime_type="application/json",
    ),
)

Uses the classification result to select a matching Pydantic schema for structured extraction.

Flatten extracted records

tax_documents["extraction"] = tax_documents.apply(extract_document, axis=1)
extracted_df = pd.json_normalize(tax_documents["extraction"])
tax_documents = tax_documents.drop(columns=["extraction"]).join(extracted_df)
tax_documents.to_csv("tax_data.csv")

Converts model-parsed structured data into a tabular CSV-ready format.

Models & APIs used

  • Models: gemini-3.5-flash
  • APIs / services: Vertex AI
  • SDKs / libraries: google-genai, pandas, pydantic

When to use this

Use this pattern when PDF tax forms need classification and typed field extraction into structured records.

Gotchas & caveats

  • Requires an existing Google Cloud project with the Vertex AI API enabled.
  • Colab authentication is only handled when running in Google Colab.
  • Defaults to GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION, with us-central1 fallback.
  • The notebook is educational only and explicitly says it is not financial advice.
  • Extraction depends on predefined Pydantic schemas for each supported document type.

Best practices

  • Use temperature=0 for deterministic classification and extraction.
  • Constrain classification with an Enum response schema.
  • Use document-specific Pydantic schemas for typed structured extraction.
  • Map classified document types to their extraction schemas before calling the model.
  • Normalize parsed JSON before joining it back into the source DataFrame.

Graph View

Backlinks