Generative AI Google Brain

search-vais-building-blocks-ingesting-unstructured-documents-with-metadata

4 min read

Ingestion of Unstructured Documents with Metadata in Vertex AI Search

Source notebook

Repo path: search/vais-building-blocks/ingesting_unstructured_documents_with_metadata.ipynb · Open on GitHub · intermediate

Ingests PDFs with metadata into Vertex AI Search and queries results with metadata filters.

Summary

This notebook shows how to prepare unstructured PDF documents and metadata for Vertex AI Search ingestion using Discovery Engine REST calls and Cloud Storage. It creates or reuses a datastore and search app, optionally patches a metadata schema, builds JSONL rows that join document URIs with structData, imports the JSONL asynchronously, and runs searches with and without a metadata filter.

Key code patterns

Authenticated REST session

from google.auth import default
from google.auth.transport.requests import AuthorizedSession
 
creds, _ = default()
authed_session = AuthorizedSession(creds)

The notebook uses Application Default Credentials and AuthorizedSession for Discovery Engine REST calls.

Create chunked datastore

payload = {
  "displayName": datastore_id,
  "solutionTypes": ["SOLUTION_TYPE_SEARCH"],
  "contentConfig": "CONTENT_REQUIRED",
  "documentProcessingConfig": {
    "chunkingConfig": {"layoutBasedChunkingConfig": {
      "chunkSize": 500,
      "includeAncestorHeadings": True}},
    "defaultParsingConfig": {"layoutParsingConfig": {}}
  }
}

Chunk mode, layout parsing, and ancestor headings are chosen in the notebook to optimize accuracy.

Create search app

payload = {
  "displayName": app_id,
  "dataStoreIds": [datastore_id],
  "solutionType": "SOLUTION_TYPE_SEARCH",
  "searchEngineConfig": {
    "searchTier": "SEARCH_TIER_ENTERPRISE",
    "searchAddOns": ["SEARCH_ADD_ON_LLM"]
  }
}

The app connects the datastore to an enterprise search engine with the LLM add-on.

Patch metadata schema

schema = {"structSchema": {"type": "object", "properties": {
  "doc_name": {"keyPropertyMapping": "title", "retrievable": True, "type": "string"},
  "quarter": {"retrievable": True, "indexable": True, "searchable": False, "type": "string"},
  "stock_tickers": {"type": "array", "items": {"type": "string", "keyPropertyMapping": "category"}}
}}}
authed_session.patch(schema_url, json=schema)

The schema controls which metadata fields are retrievable, searchable, indexable, and specially mapped.

Build JSONL import rows

def prepare_jsonl(row):
    return {
      "id": row[FIELD_FOR_FILE_NAME],
      "structData": row.to_dict(),
      "content": {
        "mimeType": "application/pdf",
        "uri": f"{GCS_DIRECTORY_DOCS}{row[FIELD_FOR_FILE_NAME]}.pdf"
      }
    }

Each JSONL record joins metadata with the GCS URI and MIME type for the unstructured document.

Import from GCS JSONL

payload = {
  "reconciliationMode": "INCREMENTAL",
  "gcsSource": {"inputUris": [gcs_uri]}
}
response = authed_session.post(
  f"{datastore_url}/branches/default_branch/documents:import",
  data=json.dumps(payload), headers={"Content-Type": "application/json"})
import_lro = response.json()["name"]

Document import is asynchronous and returns a long-running operation name to poll.

response = authed_session.post(search_url, json={
  "query": "Google revenue",
  "filter": "quarter: ANY(\"Q2\")"
})

The notebook demonstrates narrowing search results to documents whose metadata matches Q2.

Models & APIs used

  • APIs / services: Service Usage API, Cloud Storage API, Discovery Engine API, Vertex AI Search
  • SDKs / libraries: google-cloud-storage, google-auth, pandas, requests

When to use this

Use this pattern when ingesting unstructured documents from Cloud Storage into Vertex AI Search with metadata for retrieval, filtering, boosting, or returned context.

Gotchas & caveats

  • Billing and the Service Usage API, Cloud Storage API, and Discovery Engine API must be enabled.
  • The notebook expects project access through Colab authentication or Application Default Credentials.
  • Permissions listed include service usage admin, service account admin, discovery engine admin, and storage object admin roles.
  • Datastore location is chosen at creation and must be used when querying; the notebook offers global, us, and eu.
  • Datastore creation, app creation, schema update, and document import may take minutes and are polled as long-running operations.
  • The sample metadata generator assumes filenames matching patterns like 2022_Q1_Earnings_Transcript.pdf.
  • The JSONL join assumes FIELD_FOR_FILE_NAME metadata values match document filename stems.
  • Outside Colab, packages such as google.cloud.storage and google.auth may need installation.

Best practices

  • Use layout-based chunking with a 500 token chunk size and ancestor headings when accuracy on complex documents matters.
  • Provide an explicit metadata schema when fields must be retrievable, indexable, searchable, filterable, or specially mapped.
  • Represent each document as JSONL with id, structData, content.mimeType, and content.uri.
  • Poll long-running operations for schema updates and document imports before relying on the datastore.
  • Keep documents, metadata, and generated JSONL in the same temporary bucket when cleanup simplicity matters.
  • Use metadata filters such as quarter: ANY(“Q2”) to constrain search results to matching documents.

Graph View

Backlinks