Generative AI Google Brain

embeddings-vector-search-2-quickstart

4 min read

Vector Search 2.0 Public Preview Quickstart

Source notebook

Repo path: embeddings/vector-search-2-quickstart.ipynb · Open on GitHub · intermediate

Creates a Vector Search 2.0 movie collection with CRUD, filters, semantic search, and ANN indexes.

Summary

This notebook teaches the Vector Search 2.0 public preview API from setup through cleanup. It installs google-cloud-vectorsearch, authenticates, enables APIs, creates a movie collection with data and vector schemas, then populates data objects by single create, batch create, and GCS import. It demonstrates querying, filtering, aggregates, dense vector search, semantic search, text search, batch search with reciprocal rank fusion, dense and sparse ANN indexes, and deletion of resources.

Key code patterns

Initialize clients

from google.cloud import vectorsearch_v1beta
 
vector_client = vectorsearch_v1beta.VectorSearchServiceClient()
data_client = vectorsearch_v1beta.DataObjectServiceClient()
search_client = vectorsearch_v1beta.DataObjectSearchServiceClient()

The notebook separates collection/index operations, data object CRUD, and search/query calls into three clients.

Create collection schema

request = vectorsearch_v1beta.CreateCollectionRequest(
    parent=f"projects/{PROJECT_ID}/locations/{LOCATION}",
    collection_id=collection_id,
    collection={
        "data_schema": {"type": "object", "properties": {"year": {"type": "number"}, "genre": {"type": "string"}}},
        "vector_schema": {
            "plot_embedding": {"dense_vector": {"dimensions": 3}},
            "genre_embedding": {"dense_vector": {"dimensions": 4, "vertex_embedding_config": {"model_id": "text-embedding-004", "text_template": "Movie: {title} Genre: {genre} Year: {year}", "task_type": "RETRIEVAL_DOCUMENT"}}},
            "sparse_embedding": {"sparse_vector": {}},
        },
    },
)
vector_client.create_collection(request=request).result()

Defines typed metadata plus dense, sparse, and auto-generated embedding fields before loading data.

Create data object

request = vectorsearch_v1beta.CreateDataObjectRequest(
    parent=f"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}",
    data_object_id=movies[0]["id"],
    data_object={"data": movies[0]["data"], "vectors": movies[0]["vectors"]},
)
data_client.create_data_object(request=request)

Shows the basic data object shape: an id, structured data, and named vectors matching the collection schema.

Import from GCS

request = vectorsearch_v1beta.ImportDataObjectsRequest(
    name=f"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}",
    gcs_import={"contents_uri": contents_uri, "error_uri": error_uri},
)
import_lro = vector_client.import_data_objects(request)
import_lro.result()

Demonstrates bulk import from Cloud Storage and waits for the long-running operation to finish.

Filtered query

request = vectorsearch_v1beta.QueryDataObjectsRequest(
    parent=collection_name,
    filter={"$and": [{"genre": {"$eq": "Thriller"}}, {"year": {"$gte": 1995}}]},
    output_fields=vectorsearch_v1beta.OutputFields(data_fields=["*"]),
)
results = search_client.query_data_objects(request)

Uses structured filters with operators such as eq, and $gte for metadata retrieval.

request = vectorsearch_v1beta.SearchDataObjectsRequest(
    parent=collection_name,
    vector_search=vectorsearch_v1beta.VectorSearch(
        search_field="plot_embedding",
        vector=vectorsearch_v1beta.DenseVector(values=normalize([0.3, 0.4, 0.5])),
        filter={"genre": {"$eq": "Thriller"}},
        top_k=5,
        output_fields=vectorsearch_v1beta.OutputFields(data_fields=["*"]),
    ),
)
results = search_client.search_data_objects(request)

Combines dense vector similarity with metadata filters and explicit result fields.

request = vectorsearch_v1beta.SearchDataObjectsRequest(
    parent=collection_name,
    semantic_search=vectorsearch_v1beta.SemanticSearch(
        search_text="Wonderful genre of a Wonderful movie",
        search_field="genre_embedding",
        task_type="RETRIEVAL_QUERY",
        top_k=5,
        output_fields=vectorsearch_v1beta.OutputFields(data_fields=["*"]),
    ),
)
results = search_client.search_data_objects(request)

Uses the configured embedding field to search from natural language text instead of supplying a vector.

Create ANN index

request = vectorsearch_v1beta.CreateIndexRequest(
    parent=collection_name,
    index_id="plot_index",
    index={
        "index_field": "plot_embedding",
        "filter_fields": ["year", "genre"],
        "store_fields": ["title"],
    },
)
index_lro = vector_client.create_index(request)
index_lro.result()

Creates an ANN index over a vector field while preserving filter and stored fields for search results.

Models & APIs used

  • Models: text-embedding-004
  • APIs / services: Vector Search API, Vertex AI API, Cloud Storage
  • SDKs / libraries: google-cloud-vectorsearch, google-cloud-storage

When to use this

Use this pattern to prototype Vector Search 2.0 collections that need metadata CRUD, dense or sparse vectors, semantic search, text search, filters, and ANN indexes.

Gotchas & caveats

  • A Google Cloud project linked to billing is required.
  • The notebook says to request the Security Admin IAM role to enable APIs and interact with Agent Platform resources.
  • Colab requires google.colab auth.authenticate_user().
  • The setup enables vectorsearch.googleapis.com and aiplatform.googleapis.com.
  • Auto-Embeddings require the Vertex Prediction API to be enabled.
  • GCS import fails if the collection already has an ANN index.
  • The import directory must only contain import data and the error directory must be empty.
  • The GCS bucket must already exist unless the create_bucket line is uncommented.
  • Index creation operations typically take several minutes or more.
  • Collections and associated indexes should be deleted after the tutorial to avoid unexpected costs.

Best practices

  • Define data_schema and vector_schema before creating data objects.
  • Normalize generated dense vectors before storing or searching them.
  • Sort sparse embedding indices when generating sparse vectors.
  • Use batch create, batch update, batch delete, and batch search for multi-object operations.
  • Use output_fields to control returned data, vector, and metadata fields.
  • Poll long-running operations with result() before assuming imports, indexes, or deletions are complete.
  • Delete ANN indexes before cleaning up the collection.
  • Clean up data objects and the collection after the tutorial.

Graph View

Backlinks