Generative AI Google Brain

gemini-evaluation-evaltask-approach-intro-to-gen-ai-evaluation-service-sdk

2 min read

Getting Started with Vertex AI Python SDK for Gen AI Evaluation Service

Source notebook

Repo path: gemini/evaluation/evaltask_approach/intro_to_gen_ai_evaluation_service_sdk.ipynb · Open on GitHub · intro

Defines a custom pointwise metric and evaluates stored LLM responses with Vertex AI Gen AI Evaluation Service.

Summary

This notebook teaches how to use the Vertex AI Python SDK for Gen AI Evaluation Service. It installs the evaluation extra, authenticates and initializes Vertex AI, defines a custom text_quality pointwise metric with criteria and a rating rubric, evaluates a pandas dataset of stored responses with EvalTask, displays results, and deletes the created ExperimentRun.

Key code patterns

Initialize Vertex AI

PROJECT_ID = "[your-project-id]"
LOCATION = "us-central1"
EXPERIMENT_NAME = "my-eval-task-experiment"
 
vertexai.init(project=PROJECT_ID, location=LOCATION)

Evaluation runs are tied to a Google Cloud project, region, and experiment.

Define custom pointwise metric

text_quality = PointwiseMetric(
    metric="text_quality",
    metric_prompt_template=PointwiseMetricPromptTemplate(
        criteria={"fluency": "...", "entertaining": "..."},
        rating_rubric={"1": "...", "0": "...", "-1": "..."},
    ),
)

Custom criteria and rubrics define how generated text quality is judged.

Evaluate stored responses

eval_dataset = pd.DataFrame({"response": responses})
 
eval_result = EvalTask(
    dataset=eval_dataset,
    metrics=[text_quality],
    experiment=EXPERIMENT_NAME,
).evaluate()

EvalTask evaluates a dataset of model responses against the configured metrics.

Display and clean up run

notebook_utils.display_eval_result(eval_result)
 
aiplatform.ExperimentRun(
    run_name=eval_result.metadata["experiment_run"],
    experiment=eval_result.metadata["experiment"],
).delete()

The notebook shows result inspection and deletion of the created ExperimentRun.

Models & APIs used

  • APIs / services: Vertex AI, Gen AI Evaluation Service
  • SDKs / libraries: google-cloud-aiplatform[evaluation], vertexai, pandas

When to use this

Use this pattern to score stored generative AI text responses with custom rubric-based Vertex AI evaluation metrics.

Gotchas & caveats

  • The notebook uses billable Vertex AI components.
  • The runtime must restart after installing google-cloud-aiplatform[evaluation].
  • Colab requires explicit auth.authenticate_user().
  • PROJECT_ID must be set before running vertexai.init().
  • The tutorial uses LOCATION = “us-central1”.

Best practices

  • Define evaluation criteria and rating rubrics before running EvalTask.
  • Store responses in a pandas DataFrame with a response column for evaluation.
  • Inspect both summary metrics and row-based metrics in the EvalResult.
  • Delete the created ExperimentRun during cleanup.

Graph View

Backlinks