Generative AI Google Brain

search-vertexai-search-options-vertex-ai-search-website-summary

3 min read

Q&A Chatbot with Vertex AI Search for summarized website results without advanced indexing

Source notebook

Repo path: search/vertexai-search-options/vertex_ai_search_website_summary.ipynb · Open on GitHub · intermediate

Builds a Q&A flow that searches a Vertex AI Search website data store, fetches the top page, and summarizes it with Gemini.

Summary

The notebook shows how to query a website data store in Vertex AI Search through the Discovery Engine API, extract the first result link, and load page text with requests and BeautifulSoup. It then formats that web content into a prompt and calls a LangChain VertexAI model using gemini-2.0-flash to answer or summarize based on the retrieved page content.

Diagrams

search_options.png — source

Key code patterns

serving_config = client.serving_config_path(
    project=PROJECT_ID, location=LOCATION,
    data_store=DATA_STORE_ID, serving_config="default_config")
request = discoveryengine.SearchRequest(
    serving_config=serving_config,
    query=search_query,
    page_size=5,
    content_search_spec=content_search_spec)
response = client.search(request)

Queries the website data store and requests snippets from Vertex AI Search.

first_result = response.results[0]
result_json = json_format.MessageToDict(first_result.document._pb)
derived_struct_data = result_json.get("derivedStructData", {})
link = derived_struct_data.get("link", None)

Uses the first search result as the source URL for downstream page extraction.

Webpage text extraction

response = requests.get(link)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
title = soup.title.string.strip() if soup.title else "No title available"
description_meta = soup.find("meta", {"name": "description"})
page_content = " ".join(p.get_text() for p in soup.find_all("p"))

Fetches the selected page and extracts title, meta description, and paragraph text.

Gemini answer generation

chain = VertexAI(
    model_name="gemini-2.0-flash",
    generation_config={"temperature": 0.2, "max_output_tokens": 4000})
formatted_prompt = WEBPAGE_EXTRACTION_PROMPT.format(context=page_content)
response = chain(formatted_prompt)

Sends retrieved page content to Gemini through LangChain for concise Q&A output.

Models & APIs used

  • Models: gemini-2.0-flash
  • APIs / services: Vertex AI, Vertex AI Search, Discovery Engine API
  • SDKs / libraries: google-cloud-discoveryengine, langchain_google_vertexai, langchain_core, vertexai, beautifulsoup4, requests

When to use this

Use this pattern when a website data store in Vertex AI Search can identify relevant pages but you still need to fetch and summarize full page text without advanced indexing.

Gotchas & caveats

  • Colab requires google.colab auth.authenticate_user; Vertex AI Workbench does not require that step.
  • PROJECT_ID falls back to GOOGLE_CLOUD_PROJECT and LOCATION falls back to GOOGLE_CLOUD_REGION or us-central1.
  • DATA_STORE_ID must be provided for the target website data store.
  • The Discovery Engine client uses a regional endpoint unless LOCATION is global.
  • The flow assumes the first search result is the most relevant link.
  • requests.get and BeautifulSoup extraction depend on the target page being reachable and containing useful paragraph text.
  • The notebook installs google-cloud-discoveryengine==0.12.1 and restarts the kernel.

Best practices

  • Initialize Vertex AI with project and location before model use.
  • Use query expansion and spell correction with AUTO mode in SearchRequest.
  • Request snippets with ContentSearchSpec.SnippetSpec(return_snippet=True).
  • Handle empty search responses and exceptions by returning None or a fallback message.
  • Use low temperature for concise grounded answer generation.
  • Tell the model to answer only from provided page content and say when the answer is not found.

Graph View

Backlinks