Generative AI Google Brain

search-vais-building-blocks-query-level-boosting-filtering-and-facets

3 min read

Query-Level Boosting, Filtering, and Facets for Vertex AI Search Website Datastores

Source notebook

Repo path: search/vais-building-blocks/query_level_boosting_filtering_and_facets.ipynb · Open on GitHub · intermediate

Configures Vertex AI Search website datastore schema, filters, facets, and boosting via REST.

Summary

This notebook teaches how to create and query an Advanced Website datastore and search app in Vertex AI Search using Discovery Engine REST calls. It demonstrates updating schema fields from website metadata, then applying predefined-field filters, metadata filters, facet specs, and boost specs to influence retrieval and ranking. The workflow uses Google Play Books URLs as an example corpus and emphasizes testing default rankings before adding custom rules.

Key code patterns

Authenticated REST session

from google.auth import default
from google.auth.transport.requests import AuthorizedSession
 
creds, _ = default()
authed_session = AuthorizedSession(creds)

Creates an authenticated HTTP session for Discovery Engine REST API calls.

Create website datastore

payload = {
  "displayName": datastore_id,
  "industryVertical": "GENERIC",
  "solutionTypes": ["SOLUTION_TYPE_SEARCH"],
  "contentConfig": "PUBLIC_WEBSITE",
}
authed_session.post(es_endpoint, data=json.dumps(payload), headers=header)

Defines a public website datastore for Vertex AI Search indexing.

Create search app

payload = {
  "displayName": app_id,
  "dataStoreIds": [datastore_id],
  "solutionType": "SOLUTION_TYPE_SEARCH",
  "searchEngineConfig": {
    "searchTier": "SEARCH_TIER_ENTERPRISE",
    "searchAddOns": ["SEARCH_ADD_ON_LLM"]
  }
}

Connects a search app to the datastore with enterprise search and LLM add-on enabled.

Update metadata schema

json_data = {"structSchema": {"type": "object", "properties": {
  "aggregate_rating": {"type": "array", "items": {"type": "number", "retrievable": True, "indexable": True, "dynamicFacetable": True,
  "siteSearchSchemaOrgPaths": ["_root.aggregateRating.ratingValue"]}}
}}}
authed_session.patch(es_endpoint, headers=header, json=json_data)

Maps Schema.org page metadata into retrievable, indexable, and facetable fields.

Metadata filter

authed_session.post(endpoint, headers={"Content-Type": "application/json"}, json={
  "query": QUERY,
  "filter": "rating_count>10 AND aggregate_rating>4.5 AND price=0",
  "pageSize": PAGE_SIZE,
})

Restricts results using custom indexed metadata fields.

Facet specs

"facetSpecs": [
  {"facetKey": {"key": "author"}, "limit": 2, "enableDynamicPosition": False},
  {"facetKey": {"key": "aggregate_rating", "intervals": [
    {"minimum": 0, "maximum": 3}, {"minimum": 3, "maximum": 4.5}, {"minimum": 4.5, "maximum": 5}
  ]}, "limit": 3, "enableDynamicPosition": True}
]

Returns fixed and dynamic facets for search-result narrowing in a UI.

Boost spec

"boostSpec": {
  "conditionBoostSpecs": {
    "condition": "author: ANY(\"Margaret Atwood\")",
    "boost": 0.9
  }
}

Boosts matching documents without fully excluding other relevant results.

Models & APIs used

  • APIs / services: Service Usage API, Cloud Storage API, Discovery Engine API, Vertex AI Search
  • SDKs / libraries: google.auth

When to use this

Use this pattern when building Vertex AI Search website search that needs query-time filtering, facets, and ranking controls.

Gotchas & caveats

  • DATASTORE_ID and APP_ID must match the pattern [a-z0-9][a-z0-9-_]*.
  • The datastore location is set at creation time and must be used when querying.
  • Domain ownership must be verified before indexing a website.
  • Advanced website search supports up to 500 include and exclude URL patterns.
  • Datastore creation, app creation, initial indexing, and schema updates can take minutes to hours.
  • Dynamic facets require fields to be indexable and dynamicFacetable in the schema.
  • Running outside Colab may require Application Default Credentials or gcloud authentication.
  • The notebook assumes permissions such as serviceusage.serviceUsageAdmin, iam.serviceAccountAdmin, and discoveryengine.admin.

Best practices

  • Evaluate Vertex AI Search results without extra rules before incrementally adding custom filters or boosts.
  • Apply boosting and filtering during retrieval rather than post-processing search results.
  • Use metadata fields as schema hooks for filters, facets, and boost conditions.
  • Use facets to translate user preferences into subsequent filtered search requests.
  • Start with smaller boost values and adjust as needed.
  • Use user events, search tuning, custom embeddings, or synonyms where appropriate before relying on custom ranking rules.

Graph View

Backlinks