Generative AI Google Brain

embeddings-use-cases-outlier-detection-bq-vector-search-outlier-detection-audit-logs

3 min read

Log Anomaly Detection & Investigation with Text Embeddings + BigQuery Vector Search

Source notebook

Repo path: embeddings/use-cases/outlier-detection/bq-vector-search-outlier-detection-audit-logs.ipynb · Open on GitHub · advanced

Detects audit log anomalies using text-embedding-005 embeddings and BigQuery VECTOR_SEARCH.

Summary

This notebook teaches how to summarize Cloud Audit Logs into natural-language admin action records, generate text embeddings in BigQuery, and build a BigQuery vector index. It demonstrates searching historical log summaries for semantically similar actions to investigate suspicious delete operations, then visualizing nearest-neighbor distances in tables and timecharts.

Key code patterns

Summarize audit logs

CREATE OR REPLACE FUNCTION dataset.stringifyAdminLogEntry(...)
RETURNS STRING AS (
  "On " || CAST(day AS STRING) || ", principal " || principal_email ||
  " ran operation " || action || " over " || resource_type || " " || resource_id
);

Converts nested audit log fields into text suitable for embedding.

Create remote embedding model

CREATE OR REPLACE MODEL `project.dataset.embedding_model`
REMOTE WITH CONNECTION `project.US.bqdf-llm-embeddings`
OPTIONS (ENDPOINT = 'text-embedding-005');

Lets BigQuery call the embedding model through a Cloud resource connection.

Generate embeddings in BigQuery

SELECT * EXCEPT (ml_embed_text_status, statistics)
FROM ML.GENERATE_TEXT_EMBEDDING(
  MODEL `project.dataset.embedding_model`,
  TABLE `project.dataset.nb_admin_actions_summary`,
  STRUCT(TRUE AS flatten_json_output)
);

Stores embedding vectors alongside summarized log records.

Create vector index

CREATE OR REPLACE VECTOR INDEX `my_vector_index`
ON `project.dataset.nb_admin_actions_summary_embeddings`(text_embedding)
OPTIONS(distance_type='COSINE', index_type='IVF');

Indexes embeddings for managed semantic search in BigQuery.

Search similar actions

SELECT query.content, base.content, distance
FROM VECTOR_SEARCH(
  TABLE `project.dataset.nb_admin_actions_summary_embeddings`, 'text_embedding',
  TABLE `project.dataset.nb_admin_actions_summary_test_actions`,
  top_k => 5
)
WHERE query.content != base.content;

Finds historical actions semantically close to suspicious actions.

Models & APIs used

  • Models: text-embedding-005
  • APIs / services: Vertex AI, BigQuery, Cloud Logging
  • SDKs / libraries: google-cloud-aiplatform, google-cloud-bigquery, pandas, mpld3, matplotlib

When to use this

Use this pattern to investigate whether current Cloud Audit Log actions are anomalous compared with historical activity.

Gotchas & caveats

  • Requires upgrading the log bucket to Log Analytics to get a linked BigQuery dataset.
  • Requires BigQuery Data Viewer on the source linked dataset.
  • Requires a BigQuery Cloud resource connection and its service account granted roles/aiplatform.user.
  • Embedding generation may take several minutes and incur Agent Platform costs depending on row count.
  • Vector index storage is chargeable and index status should be checked in INFORMATION_SCHEMA.VECTOR_INDEXES.

Best practices

  • Aggregate daily user actions before embedding to make log analysis easier, faster, and more cost-effective.
  • Include principal, action, resource, container, channel, IP, and count in the text used for embeddings.
  • Normalize resource IDs and user-agent channels with BigQuery UDFs before embedding.
  • Remove exact duplicate matches when comparing suspicious actions to historical actions.
  • Use distance thresholds and visualizations to support anomaly investigation.

Graph View

Backlinks