Generative AI Google Brain

embeddings-vector-search-2-intro

3 min read

Introduction to Agent Platform Vector Search 2.0

Source notebook

Repo path: embeddings/vector-search-2-intro.ipynb · Open on GitHub · intermediate

Builds an e-commerce product search demo with Agent Platform Vector Search 2.0 and auto-embeddings.

Summary

The notebook introduces Agent Platform Vector Search 2.0 collections, data objects, querying, filtering, semantic search, text search, and hybrid search. It creates a product collection for TheLook e-commerce data, configures auto-generated dense embeddings from product names, imports a 10,000 product sample, and searches products with filters and Reciprocal Rank Fusion. It contrasts kNN for immediate development search with ANN indexes for production-scale latency.

Key code patterns

Create clients

from google.cloud import vectorsearch_v1beta
 
vector_search_service_client = vectorsearch_v1beta.VectorSearchServiceClient()
data_object_service_client = vectorsearch_v1beta.DataObjectServiceClient()
data_object_search_service_client = vectorsearch_v1beta.DataObjectSearchServiceClient()

Separates collection/index management, data object writes, and search/query operations.

Collection with auto-embeddings

request = vectorsearch_v1beta.CreateCollectionRequest(
    parent=f"projects/{PROJECT_ID}/locations/{LOCATION}",
    collection_id=collection_id,
    collection={"data_schema": {...}, "vector_schema": {
        "name_dense_embedding": {"dense_vector": {
            "dimensions": 768,
            "vertex_embedding_config": {
                "model_id": "gemini-embedding-001",
                "text_template": "{name}",
                "task_type": "RETRIEVAL_DOCUMENT"}}}}})

Defines product fields and lets the service generate dense embeddings from product names.

Batch import objects

batch_size = 250
for batch_start in range(0, len(products), batch_size):
    request = vectorsearch_v1beta.BatchCreateDataObjectsRequest(
        parent=f"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}",
        requests=[{"data_object_id": p["id"], "data_object": {"data": p["data"], "vectors": {}}}
                  for p in products[batch_start:batch_start + batch_size]],
    )
    data_object_service_client.batch_create_data_objects(request)

Uses empty vectors to trigger auto-embedding generation while respecting the 250 texts per request limit.

request = vectorsearch_v1beta.SearchDataObjectsRequest(
    parent=f"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}",
    semantic_search=vectorsearch_v1beta.SemanticSearch(
        search_text="outfit for beach",
        search_field="name_dense_embedding",
        task_type="QUESTION_ANSWERING",
        top_k=10,
        filter={"$and": [{"category": {"$eq": "Shorts"}}, {"retail_price": {"$lt": 30}}]},
    ),
)

Combines natural language search intent with exact category and price constraints.

Hybrid search with RRF

request = vectorsearch_v1beta.BatchSearchDataObjectsRequest(
    parent=f"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}",
    searches=[vectorsearch_v1beta.Search(semantic_search=...),
              vectorsearch_v1beta.Search(text_search=...)],
    combine=vectorsearch_v1beta.BatchSearchDataObjectsRequest.CombineResultsOptions(
        ranker=vectorsearch_v1beta.Ranker(
            rrf=vectorsearch_v1beta.ReciprocalRankFusion(weights=[1.0, 1.0]))),
)

Merges semantic relevance and keyword precision into one ranked result list.

Models & APIs used

  • Models: gemini-embedding-001
  • APIs / services: Vector Search API, Agent Platform API, Vertex AI
  • SDKs / libraries: google-cloud-vectorsearch, google.cloud.vectorsearch_v1beta, tqdm

When to use this

Use this pattern when building filtered semantic, keyword, or hybrid product search over catalog data in Google Cloud.

Gotchas & caveats

  • A Google Cloud project must be linked to a billing account.
  • The setup enables vectorsearch.googleapis.com and aiplatform.googleapis.com.
  • Colab requires google.colab auth.authenticate_user(); Colab Enterprise and Workbench can skip it.
  • The tutorial uses LOCATION = “us-central1”.
  • Vector Search 2.0 resources incur costs when active and should be cleaned up.
  • Batch size must not exceed the embedding model max texts per request, 250 for gemini-embedding-001.
  • Auto-embeddings are subject to Agent Platform Embeddings API quotas, noted as 5M tokens/min and 250 texts/request.
  • The data schema note says additionalProperties=True is not currently supported.

Best practices

  • Use a schema-enforced collection with separate data_schema and vector_schema.
  • Use auto-embeddings by leaving vectors empty when vertex_embedding_config is configured.
  • Use random.seed(42) for reproducible sampling of the product dataset.
  • Use batch_create_data_objects instead of one create request per product for bulk imports.
  • Use task_type=“RETRIEVAL_DOCUMENT” for indexed product names and task_type=“QUESTION_ANSWERING” for semantic queries.
  • Use kNN for immediate development search and ANN indexes for large production datasets.
  • Apply filters to both semantic and text searches in hybrid search so combined results obey the same constraints.
  • Run the cleanup section to delete Collections and Indexes after the tutorial.

Graph View

Backlinks