Generative AI Google Brain

gemini-evaluation-evaluating-observability-datasets

2 min read

Using Gen AI Evaluation SDK for Google Observability Gen AI multi-modal datasets

Source notebook

Repo path: gemini/evaluation/evaluating_observability_datasets.ipynb · Open on GitHub · intermediate

Evaluates Google Observability multimodal Gen AI data from GCS using Vertex AI Gen AI Evaluation.

Summary

This notebook shows how to evaluate Gen AI prompt, response, and system instruction data stored by Google Observability in Google Cloud Storage. It installs the Vertex AI SDK evaluation extra, authenticates when running in Colab, initializes a Vertex AI client, loads ObservabilityEvalCase references into an EvaluationDataset, and runs client.evals.evaluate with the default GENERAL_QUALITY metric.

Key code patterns

Initialize Vertex AI client

from vertexai import Client, types
 
client = Client(project=PROJECT_ID, location=LOCATION)

Creates the SDK client used to load observability data and run evaluations.

Load observability eval dataset

eval_case = types.ObservabilityEvalCase(
    input_src=INPUT_SOURCE,
    output_src=OUTPUT_SOURCE,
    system_instruction_src=SYSTEM_INSTRUCTION_SOURCE,
    api_client=client,
)
eval_dataset = types.EvaluationDataset.load_from_observability_eval_cases([eval_case])

Maps separate GCS references for prompt, response, and system instruction data into an evaluation dataset.

Run evaluation

eval_result = client.evals.evaluate(dataset=eval_dataset)
 
eval_result.show()

Runs the default GENERAL_QUALITY adaptive rubric-based evaluation and displays results.

Models & APIs used

  • APIs / services: Vertex AI, Google Cloud Storage, Google Observability
  • SDKs / libraries: google-cloud-aiplatform, vertexai

When to use this

Use this pattern when Gen AI observability traces store prompt, response, and system instruction content in GCS and you need Vertex AI evaluation over that data.

Gotchas & caveats

  • Requires google-cloud-aiplatform[evaluation]>=1.122.0.
  • Colab users must authenticate with google.colab.auth.authenticate_user().
  • A Google Cloud project is required and the Vertex AI API must be enabled.
  • INPUT_SOURCE, OUTPUT_SOURCE, and SYSTEM_INSTRUCTION_SOURCE must point to the observability GCS references.
  • Default location is us-central1 unless GOOGLE_CLOUD_REGION overrides it.

Best practices

  • Use environment variables GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION as fallbacks for notebook parameters.
  • Keep prompt, response, and system instruction sources explicit when constructing ObservabilityEvalCase.
  • Call show() on the loaded dataset and evaluation result to inspect inputs and outputs.

Graph View

Backlinks