Generative AI Google Brain

embeddings-use-cases-outlier-detection-bq-vector-search-outlier-detection-infra-logs

3 min read

Anomaly Detection of Infrastructure Logs using Gemini and BigQuery Vector Search

Source notebook

Repo path: embeddings/use-cases/outlier-detection/bq-vector-search-outlier-detection-infra-logs.ipynb · Open on GitHub · advanced

Detects HDFS log anomalies with Gemini summaries, text embeddings, and BigQuery Vector Search.

Summary

This notebook shows how to ingest parsed HDFS infrastructure logs into BigQuery, group them by block ID, label sessions, and summarize each log sequence with Gemini from BigQuery ML. It generates text embeddings for the Gemini summaries, stores them in BigQuery, builds a TREE_AH vector index, visualizes clusters with t-SNE, and uses VECTOR_SEARCH SQL to classify outliers. It also frames recall as the primary metric and compares the vector-search approach with unsupervised and semi-supervised anomaly detection methods.

Key code patterns

BigQuery remote Gemini model

CREATE OR REPLACE MODEL `vs_logs_demo.gemini_1_5_flash`
REMOTE WITH CONNECTION `us.bq-llm-connection`
OPTIONS (endpoint = 'gemini-2.0-flash')

Lets BigQuery call Gemini through a cloud resource connection.

Generate log summaries

FROM ML.GENERATE_TEXT(
  MODEL `vs_logs_demo.gemini_1_5_flash`,
  (SELECT CONCAT(prompt_prefix, eventSequence) AS prompt FROM ...),
  STRUCT(0.2 AS temperature, 2048 AS max_output_tokens, FALSE AS flatten_json_output)
)

Turns raw chronological HDFS log sessions into concise natural-language event summaries.

Embedding generation

FROM ML.GENERATE_EMBEDDING(
  MODEL `vs_logs_demo.text_embedding`,
  (SELECT response AS content FROM `vs_logs_demo.hdfs_full_explained` WHERE status=''),
  STRUCT(TRUE AS flatten_json_output, 'CLUSTERING' AS task_type)
)

Creates embeddings optimized for clustering-based anomaly detection.

Vector index

CREATE VECTOR INDEX index_treeah_1000
ON `vs_logs_demo.hdfs_full_embeddings`(embeddings)
OPTIONS (index_type = 'TREE_AH', distance_type = 'COSINE', tree_ah_options = '{"leaf_node_embedding_count": 1000}')

Builds an indexed vector store in BigQuery for fast nearest-neighbor lookup.

SELECT query.blockId,
  COUNTIF(distance < threshold) AS similar_normal_instances,
  IF(COUNTIF(distance < threshold) >= n_neighbors, 0, 1) AS predicted
FROM VECTOR_SEARCH(base_table, 'embeddings', TABLE test_table, top_k => n_neighbors, distance_type => 'COSINE', options => '{"use_brute_force":true}')

Flags sessions as anomalous when they lack enough nearby normal examples.

Models & APIs used

  • Models: gemini-2.0-flash, text-embedding-005
  • APIs / services: Vertex AI, BigQuery, BigQuery ML, BigQuery Vector Search, Cloud Storage
  • SDKs / libraries: google-cloud-aiplatform, google-cloud-bigquery, bigframes.pandas, scikit-learn

When to use this

Use this pattern to detect novel anomalies in large infrastructure log datasets where normal behavior can be represented by semantic embeddings.

Gotchas & caveats

  • Requires a BigQuery cloud resource connection and its service account needs roles/aiplatform.user.
  • The notebook uses us-central1 for Vertex AI and US for BigQuery location.
  • Gemini summarization over 70k sessions may take up to 6 hours depending on quota.
  • Default noted quota for gemini-2.0-flash is 200 requests per minute in us-central1 at the time of writing.
  • BigQuery ML ML.GENERATE_TEXT row limits require batching around 20k rows for this workflow.
  • ML.GENERATE_EMBEDDING is run as one job because the notebook states a 2,700,000 rows per job limit.
  • TREE_AH prefilters on the base table are treated as post-filters and may reduce top-k results.
  • Runtime restart is required after installing packages.

Best practices

  • Group raw logs by a meaningful session identifier before summarization.
  • Use a one-shot prompt and instruct Gemini not to repeat raw logs or speculate.
  • Persist intermediate outputs in BigQuery tables so long-running jobs can resume after failures.
  • Batch ML.GENERATE_TEXT calls to reduce loss from timeouts or quota exhaustion.
  • Use CLUSTERING task_type for embeddings when the downstream task is clustering-based anomaly detection.
  • Split training and test chronologically and train only on normal sessions for novelty detection.
  • Use brute force VECTOR_SEARCH when maximizing recall for outlier detection evaluation.
  • Inspect VECTOR_INDEXES to confirm vector index coverage and storage.

Graph View

Backlinks