Generative AI Google Brain

search-vais-building-blocks-custom-attributes-by-url-pattern

3 min read

Defining custom attributes based on URL patterns in Vertex AI Search Website Datastores

Source notebook

Repo path: search/vais-building-blocks/custom_attributes_by_url_pattern.ipynb · Open on GitHub · intermediate

Shows how to add URL-pattern custom attributes to a Vertex AI Search website datastore.

Summary

This notebook demonstrates creating or reusing a Vertex AI Search advanced website datastore and search app, then configuring URL include patterns. It sets custom attributes from URL patterns through Discovery Engine REST endpoints, verifies the mapping, and searches with and without a metadata filter. It also includes cleanup calls for deleting the app and datastore.

Key code patterns

Authenticated REST session

from google.auth import default
from google.auth.transport.requests import AuthorizedSession
 
creds, _ = default()
authed_session = AuthorizedSession(creds)

Uses Application Default Credentials to call Discovery Engine REST endpoints.

Create website datastore

payload = {
    "displayName": datastore_id,
    "industryVertical": "GENERIC",
    "solutionTypes": ["SOLUTION_TYPE_SEARCH"],
    "contentConfig": "PUBLIC_WEBSITE",
}
authed_session.post(es_endpoint, data=json.dumps(payload), headers=header)

Creates a public website datastore for Vertex AI Search.

Set URL attribute mapping

json_data = {
    "documentDataMap": {
        "https://cloud.google.com/generative-ai-app-builder/docs/reference/rest/v1alpha/*": {
            "Topic": ["Rest", "V1alpha"]
        }
    },
    "schema": {"properties": {"Topic": {"type": "array"}}, "type": "object"},
}
authed_session.post(es_endpoint, headers=header, json=json_data)

Maps URL patterns to custom attributes using setUriPatternDocumentData.

Filter by custom attribute

authed_session.post(
    search_url,
    headers={"Content-Type": "application/json"},
    json={"query": QUERY, "filter": 'Topic: ANY("V1alpha")', "pageSize": PAGE_SIZE},
)

Uses an indexable custom attribute to restrict search results.

Models & APIs used

  • APIs / services: Vertex AI Search, Discovery Engine API, Service Usage API, Cloud Storage API
  • SDKs / libraries: google.auth

When to use this

Use this pattern when website page metadata is unavailable, private, or needs URL-based override in Vertex AI Search.

Gotchas & caveats

  • The notebook requires Google Cloud authentication through Colab auth or Application Default Credentials.
  • The project must have billing enabled and Service Usage, Cloud Storage, and Discovery Engine APIs enabled.
  • Required roles include serviceusage.serviceUsageAdmin, iam.serviceAccountAdmin, and discoveryengine.admin unless using Owner.
  • Datastore location is set at creation time; global is recommended unless a regional datastore is needed.
  • URL mapping for custom attributes is stated as available only in v1alpha at the time of writing.
  • Domain ownership must be verified before indexing a website.
  • Advanced website search supports up to 500 include and exclude URL patterns.
  • Each successful custom attribute mapping request overrides previous mappings.

Best practices

  • Use global datastore location unless there is a specific reason to use us or eu.
  • Verify schema and URL mapping with getUriPatternDocumentData after setting it.
  • Make custom attributes indexable, retrievable, and searchable when they should affect retrieval and appear in responses.
  • Poll datastore and app availability because creation can take a few minutes.
  • Delete the app and datastore when no longer needed.

Graph View

Backlinks