Generative AI Google Brain

open-models-evaluation-evaluate-open-models-byod

2 min read

Qwen 3 evaluation - Bring your own data eval

Source notebook

Repo path: open-models/evaluation/evaluate_open_models_byod.ipynb · Open on GitHub · intermediate

Compares fine-tuned and base Qwen 3 medical summaries using Vertex AI pairwise evaluation.

Summary

This notebook demonstrates bring-your-own-data evaluation with Vertex AI Evaluation service. It builds a pandas DataFrame containing prompts, candidate responses, and baseline responses for medical note summarization. It configures an EvalTask with pairwise_summarization_quality, runs evaluation, reviews the metrics table, and optionally deletes the experiment.

Key code patterns

Initialize Vertex AI

PROJECT_ID = "[your-project-id]"
LOCATION = "us-central1"
 
import vertexai
vertexai.init(project=PROJECT_ID, location=LOCATION)

Sets the Google Cloud project and region before using Vertex AI evaluation.

Build pairwise dataset

eval_dataset = pd.DataFrame({
    "prompt": prompts,
    "response": qwen_tuned_responses,
    "baseline_model_response": qwen_responses,
})

Structures BYOD evaluation rows with the prompt, candidate output, and baseline output.

Configure EvalTask

summarization_eval_task = EvalTask(
    dataset=eval_dataset,
    metrics=["pairwise_summarization_quality"],
    experiment="eval-qwen3",
)

Defines a pairwise summarization evaluation and logs it under a Vertex AI Experiment.

Run and inspect evaluation

eval_result = summarization_eval_task.evaluate()
eval_result.metrics_table

Runs the managed evaluation and displays aggregated win/loss/tie metrics.

Optional cleanup

if delete_experiment:
    experiment = aiplatform.Experiment(EXPERIMENT_NAME)
    experiment.delete()

Shows how to remove the Vertex AI Experiment when cleanup is enabled.

Models & APIs used

  • APIs / services: Vertex AI, Vertex AI Evaluation service, Vertex AI Experiments
  • SDKs / libraries: google-cloud-aiplatform, vertexai, pandas

When to use this

Use this pattern to compare candidate and baseline model outputs on a custom task-specific dataset.

Gotchas & caveats

  • Requires google-cloud-aiplatform[evaluation] to be installed.
  • Requires authentication in Colab.
  • Requires an existing Google Cloud project with the Vertex AI API enabled.
  • The notebook initializes Vertex AI in us-central1.
  • Dataset columns must include prompt, response, and baseline_model_response for this pairwise setup.
  • The model responses are precomputed lists; the notebook does not call Qwen models directly.

Best practices

  • Use a custom dataset for domain-specific model assessment.
  • Compare a candidate response against a baseline_model_response row by row.
  • Use pairwise_summarization_quality for summary quality comparison.
  • Log evaluation runs to a Vertex AI Experiment for tracking and comparison.
  • Optionally clean up experiments after running evaluations.

Graph View

Backlinks