Generative AI Google Brain

gemini-use-cases-document-processing-patents-understanding

3 min read

Patents Document Understanding with Gemini

Source notebook

Repo path: gemini/use-cases/document-processing/patents_understanding.ipynb · Open on GitHub · intermediate

Uses Gemini batch prediction on Vertex AI to extract structured fields and figure boxes from patent PDFs.

Summary

This notebook teaches how to replace a multi-model AutoML patent document pipeline with one Gemini request per PDF. It queries patent PDF Cloud Storage URIs from BigQuery, builds controlled-generation JSON requests with a schema, writes those requests to BigQuery, runs a Vertex AI Gemini batch job, flattens the structured responses, and compares them with public ground-truth patent tables.

Key code patterns

Create GenAI and BigQuery clients

from google import genai
from google.cloud import bigquery
 
PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")
LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "us-central1")
client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)
bq_client = bigquery.Client()

Initializes Vertex AI Gemini access through Google GenAI SDK and BigQuery access for source and output tables.

Fetch patent PDF URIs

query = """
SELECT COALESCE(t1.gcs_path, t2.gcs_path, t3.gcs_path) AS gcs_path
FROM `bigquery-public-data.labeled_patents.extracted_data` AS t1
FULL OUTER JOIN `bigquery-public-data.labeled_patents.figures` AS t2
ON t1.gcs_path = t2.gcs_path
FULL OUTER JOIN `bigquery-public-data.labeled_patents.invention_types` AS t3
ON COALESCE(t1.gcs_path, t2.gcs_path) = t3.gcs_path
LIMIT 5
"""
df = bq_client.query(query).result().to_dataframe()

Uses BigQuery public patent data to collect Cloud Storage PDF paths for batch processing.

Build controlled JSON request

def create_request_json(row):
    return json.dumps({
        "contents": [{"role": "user", "parts": [
            {"text": PATENTS_PROMPT},
            {"fileData": {"fileUri": row["gcs_path"], "mimeType": "application/pdf"}}
        ]}],
        "generationConfig": {
            "responseMimeType": "application/json",
            "responseSchema": response_schema
        }
    })

Combines the PDF input and prompt with a response schema so Gemini returns structured JSON.

Run Gemini batch job from BigQuery

pandas_gbq.to_gbq(df, f"{DATASET_NAME}.{TABLE_NAME}", project_id=PROJECT_ID)
 
batch_job = client.batches.create(
    model=MODEL_ID,
    src=f"bq://{PROJECT_ID}.{DATASET_NAME}.{TABLE_NAME}",
)

Loads GenerateContentRequest JSON rows into BigQuery and submits them as a Vertex AI batch prediction job.

Poll and flatten results

while batch_job.state == "JOB_STATE_RUNNING":
    batch_job = client.batches.get(name=batch_job.name)
    time.sleep(5)
 
results_df = pandas_gbq.read_gbq(
    batch_job.dest.bigquery_uri.replace("bq://", ""), project_id=PROJECT_ID
)
results_df = results_df.join(pd.json_normalize(results_df["response"].apply(flatten_response)))

Waits for batch completion, reads the destination BigQuery table, parses candidate text JSON, and expands fields into columns.

Models & APIs used

  • Models: gemini-3.5-flash
  • APIs / services: Vertex AI, BigQuery, Cloud Storage
  • SDKs / libraries: google-genai, google-cloud-bigquery, pandas-gbq

When to use this

Use this pattern when processing many PDF documents with Gemini and storing structured extraction results in BigQuery.

Gotchas & caveats

  • The Vertex AI API must be enabled for the Google Cloud project.
  • Colab users must authenticate with google.colab.auth.authenticate_user().
  • The notebook installs packages then restarts the runtime before continuing.
  • The BigQuery dataset location must match the Vertex AI location, such as us-central1 instead of us.
  • The tutorial limits the source query to 5 documents to reduce processing time.
  • Batch job results are read from the job destination BigQuery URI.
  • Response parsing can fail on missing keys, empty candidates, or invalid JSON, so flatten_response catches KeyError, IndexError, and JSONDecodeError.

Best practices

  • Use a detailed prompt plus responseMimeType application/json and responseSchema for controlled generation.
  • Include the PDF as fileData with mimeType application/pdf instead of extracting document text separately.
  • Use a systemInstruction to set the model role as an expert at analyzing patent documents.
  • Write batch requests to BigQuery and let the batch job return outputs to a BigQuery table.
  • Poll batch job state before reading destination results.
  • Compare generated outputs with ground truth from the public labeled_patents tables.

Graph View

Backlinks