search-vais-building-blocks-parsing-and-chunking-with-byo

Parsing and Chunking in Vertex AI Search: Featuring BYO Capabilities

Source notebook

Repo path: search/vais-building-blocks/parsing_and_chunking_with_BYO.ipynb · Open on GitHub · intermediate

Retrieves, reviews, exports, and reimports Vertex AI Search parsed and chunked documents with BYOC.

Summary

This notebook demonstrates how to create a Vertex AI Search chunk-mode datastore, import a sample PDF from Cloud Storage, and retrieve parsed and chunked document representations through Discovery Engine REST APIs. It reconstructs chunks for visual inspection, uploads chunk JSON to Cloud Storage for offline review or editing, and shows the private-preview Bring Your Own Chunks import flow.

Key code patterns

Create chunk-mode datastore

payload = {
  "displayName": datastore_id,
  "documentProcessingConfig": {
    "chunkingConfig": {"layoutBasedChunkingConfig": {
      "chunkSize": 500,
      "includeAncestorHeadings": True}},
    "defaultParsingConfig": {"layoutParsingConfig": {}}
  }
}
authed_session.post(es_endpoint, data=json.dumps(payload), headers=header)

Configures layout-based chunking with 500-token chunks and ancestor headings.

Import PDFs from GCS

payload = {
  "reconciliationMode": "INCREMENTAL",
  "gcsSource": {
    "inputUris": [gcs_uri],
    "dataSchema": "content"
  }
}
response = authed_session.post(es_endpoint, data=json.dumps(payload), headers=header)
return response.json()["name"]

Uses content schema so unstructured PDF files can be ingested directly.

Retrieve processed document

url = f"{base_url}/projects/{project_id}/locations/global/collections/default_collection/dataStores/{data_store_id}/branches/0/documents/{document_id}:getProcessedDocument?processed_document_type=CHUNKED_DOCUMENT"
response = authed_session.get(url)
chunked_document = parse_and_print_json(response.json())

Calls getProcessedDocument to fetch parsed or chunked JSON for a datastore document.

Reconstruct chunk text

reconstructed_document = ""
for chunk in chunked_document["jsonData"]["chunks"]:
    reconstructed_document += "Start of chunk: " + chunk["id"] + "\n\n"
    reconstructed_document += chunk["content"]
    reconstructed_document += "\n\nEnd of chunk: " + chunk["id"] + "\n\n"

Stacks chunk content with IDs so engineers can visually review chunk boundaries.

Import BYO chunks

payload = {
  "reconciliationMode": "INCREMENTAL",
  "gcsSource": {
    "inputUris": uri_path,
    "dataSchema": "content"
  }
}
response = authed_session.post(url=url, json=payload)

Reimports a JSON chunk file whose format matches the chunked document jsonData field.

Models & APIs used

• APIs / services: Service Usage API, Discovery Engine API, Cloud Storage API, Vertex AI Search
• SDKs / libraries: google-auth, google-cloud-storage, requests

When to use this

Use this pattern when you need to inspect Vertex AI Search document parsing and chunking or ingest edited custom chunks into a datastore.

Gotchas & caveats

• BYOC is private preview in the notebook and requires allowlisting through a Google account team.
• Datastore location is set at creation time and must be called appropriately; examples use global endpoints for processed documents.
• Chunk mode must be enabled on the datastore before getProcessedDocument can return parsed or chunked documents.
• Imported documents are processed asynchronously through long-running operations and may take minutes.
• Chunks should comply with the datastore chunk token limit specified at creation time.
• Direct PDF ingestion requires GCS source dataSchema set to content; the default document schema expects JSONL.
• A BYOC JSON document is treated separately from the original source document, so the original may need removal to avoid duplicates.

Best practices

• Create the datastore with layout parser for complex documents containing tables and lists.
• Include ancestor headings with chunks to preserve heading context.
• Use incremental reconciliation for document and chunk imports.
• Poll datastore creation and import long-running operations before retrieving processed documents.
• Export chunked JSON to Cloud Storage for offline review or editing before BYOC import.
• Set DELETE_RESOURCES to True only when intentionally deleting the demo bucket and datastore.

Related

• Concepts: Vertex AI Search · RAG & Grounding · Applied Use Cases
• Entities: Cloud Storage · Grounding
• Area: Vertex AI Search Notebooks
• Best practices: Vertex AI Search - Best Practices · RAG & Grounding - Best Practices · Applied Use Cases - Best Practices