Generative AI Google Brain

search-vais-building-blocks-manual-recrawl-urls-with-trigger

3 min read

Event-based Triggering of Manual Recrawl for Vertex AI Search Advanced Website Datastores

Source notebook

Repo path: search/vais-building-blocks/manual_recrawl_urls_with_trigger.ipynb · Open on GitHub · intermediate

Automates Vertex AI Search manual recrawl from JSON URL lists uploaded to Cloud Storage.

Summary

It teaches how to stage URL lists in a Cloud Storage bucket, trigger a Cloud Run Function on object finalization, parse a JSON uris list, and call the Discovery Engine recrawlUris endpoint. The guide covers project/API setup, required IAM roles, function source/dependencies, deployment, and testing with a sample JSON file for an existing advanced website search datastore.

Key code patterns

Filter finalized JSON uploads

@functions_framework.cloud_event
def recrawl_uris(cloud_event):
    data = cloud_event.data
    bucket_name = data["bucket"]
    file_name = data["name"]
    if file_name.endswith(".json") and cloud_event["type"] == "google.cloud.storage.object.v1.finalized":
        uris = read_uris_from_gcs(bucket_name, file_name)
        if uris:
            recrawl_uris_with_api(uris)

Ensures the function only processes finalized Cloud Storage JSON objects.

Read URIs from GCS JSON

storage_client = storage.Client()
bucket = storage_client.bucket(bucket_name)
blob = bucket.blob(file_name)
file_content = blob.download_as_text()
data = json.loads(file_content)
return data.get("uris", [])

Loads the uploaded file and extracts the uris array expected by the recrawl request.

Call recrawlUris REST endpoint

creds, _ = default()
auth_req = GoogleAuthRequest()
creds.refresh(auth_req)
url = f"https://discoveryengine.googleapis.com/v1alpha/projects/{PROJECT_ID}/locations/global/collections/default_collection/dataStores/{DATA_STORE_ID}/siteSearchEngine:recrawlUris"
headers = {"Authorization": f"Bearer {creds.token}", "Content-Type": "application/json", "X-Goog-User-Project": PROJECT_ID}
requests.post(url, headers=headers, json={"uris": uris}, timeout=10)

Uses application default credentials to invoke the Discovery Engine manual recrawl API.

Retry transient request failures

for attempt in range(3):
    try:
        response = requests.post(url, headers=headers, json=data, timeout=10)
        response.raise_for_status()
        return
    except requests.exceptions.RequestException as e:
        if attempt < 2 and isinstance(e, (requests.exceptions.ConnectionError, requests.exceptions.Timeout)):
            time.sleep(5)

Retries only connection and timeout failures, while raising for bad HTTP status codes.

Models & APIs used

  • APIs / services: Vertex AI Search, Discovery Engine API, Cloud Storage API, Service Usage API, Cloud Run Functions
  • SDKs / libraries: functions-framework, google-cloud-storage, google-auth, requests

When to use this

Use this pattern when an existing Vertex AI Search advanced website datastore needs event-driven manual recrawls from explicit URL lists.

Gotchas & caveats

  • The notebook is not self-contained and requires an existing advanced website search datastore.
  • PROJECT_ID and DATA_STORE_ID must be replaced, and the notebook recommends environment variables for them.
  • Manual refresh recrawls explicit URLs, not URL patterns, and is subject to documented recrawl limits.
  • The function service account needs permissions such as Storage Object Viewer and Discovery Engine Admin plus invocation, service account, and logging roles listed in the notebook.
  • The sample uses us-central1 for the bucket and function example, while the recrawl API path uses locations/global.
  • The sample retry block calls time.sleep(5) but does not import time.

Best practices

  • Use a Cloud Storage staging bucket for JSON files containing URLs to recrawl.
  • Trigger processing on google.cloud.storage.object.v1.finalized events and filter for .json files.
  • Use functions-framework, described as the recommended way at the time of the notebook.
  • Fetch an access token with google-auth default credentials before calling the REST API.
  • Set a request timeout and retry connection or timeout failures up to three times.
  • Check Cloud Run Functions logs after uploading a sample JSON test file.

Graph View

Backlinks