Generative AI Google Brain

search-vais-building-blocks-inline-ingestion-of-documents

3 min read

Inline Ingestion of Documents into Vertex AI Search

Source notebook

Repo path: search/vais-building-blocks/inline_ingestion_of_documents.ipynb · Open on GitHub · intermediate

Shows inline rawBytes document ingestion, document operations, search, filters, and cleanup in Vertex AI Search.

Summary

This notebook demonstrates how to create a Vertex AI Search datastore, prepare local PDF and text documents, encode them as Base64 rawBytes, and import them inline through the Discovery Engine REST API. It then lists, retrieves, optionally deletes, searches, filters by metadata, and cleans up local files or the datastore. It contrasts inline ingestion with GCS or BigQuery staging and notes that inline ingestion is simpler but has stricter file-size limits and lower UI visibility.

Key code patterns

Authenticated REST session

from google.auth import default
from google.auth.transport.requests import AuthorizedSession
 
creds, _ = default()
authed_session = AuthorizedSession(creds)

Uses Application Default Credentials to call Discovery Engine REST endpoints.

Create VAIS datastore

payload = {
  "displayName": datastore_id,
  "industryVertical": "GENERIC",
  "solutionTypes": ["SOLUTION_TYPE_SEARCH"],
  "contentConfig": "CONTENT_REQUIRED",
  "documentProcessingConfig": {"defaultParsingConfig": {"digitalParsingConfig": {}}},
}
authed_session.post(es_endpoint, data=json.dumps(payload), headers=header)

Creates a document-mode search datastore with the basic digital parser.

Inline rawBytes import

payload = {
  "reconciliationMode": "INCREMENTAL",
  "inlineSource": {"documents": [my_document_1, my_document_2, my_document_3]},
}
authed_session.post(
  f"https://discoveryengine.googleapis.com/v1/projects/{project_id}/locations/{location}/collections/default_collection/dataStores/{datastore_id}/branches/default_branch/documents:import",
  data=json.dumps(payload), headers={"Content-Type": "application/json"})

Imports documents inline instead of referencing staged GCS or BigQuery URIs.

Base64 document content

with open(file_path, "rb") as file:
    file_data = file.read()
    base64_encoded_data = base64.b64encode(file_data).decode("utf-8")
 
my_document = {
  "id": "doc-1",
  "structData": {"title": "test_doc_1", "color_theme": "blue"},
  "content": {"mimeType": "application/pdf", "rawBytes": base64_encoded_data},
}

VAIS inline ingestion uses content.rawBytes plus an accurate mimeType.

response = authed_session.post(
  f"https://discoveryengine.googleapis.com/v1alpha/projects/{PROJECT_ID}/locations/{LOCATION}/collections/default_collection/dataStores/{DATASTORE_ID}/servingConfigs/default_search:search",
  headers={"Content-Type": "application/json"},
  json={"query": "Google revenue", "filter": 'color_theme: ANY("red")'},
)

Shows how structData metadata can constrain search results.

Models & APIs used

  • APIs / services: Service Usage API, Cloud Storage API, Discovery Engine, Vertex AI Search
  • SDKs / libraries: google.auth, requests

When to use this

Use this pattern when documents must be ingested directly into Vertex AI Search without staging files in Cloud Storage or BigQuery.

Gotchas & caveats

  • Inline ingestion has stricter file-size limits than staged imports.
  • Inline content is encoded into rawBytes, reducing visibility in the UI.
  • Datastore location is chosen at creation and must be used when querying it.
  • The notebook requires billing, enabled Service Usage, Cloud Storage, and Discovery Engine APIs.
  • Recommended permissions include serviceusage.serviceUsageAdmin, iam.serviceAccountAdmin, discoveryengine.admin, and storage.objectAdmin.
  • Datastore creation may take a few minutes and is polled with 30-second waits.
  • For non-global locations, document list/get/delete use a location-prefixed Discovery Engine host.
  • A Full import from a staged GCS or BigQuery source can make deleted documents reappear.

Best practices

  • Use GCS buckets or BigQuery tables as a source of truth when importing structured or unstructured documents unless inline ingestion is required.
  • Use Incremental or Full reconciliation modes depending on the import strategy and conflict-resolution needs.
  • Set mimeType consistently with the source file format, such as application/pdf or text/plain.
  • Add structData metadata when it should influence search filtering.
  • Poll datastore availability after creation before continuing.
  • Use DELETE_RESOURCES as a guard before deleting local files or the datastore.

Graph View

Backlinks