Generative AI Google Brain

gemini-evaluation-evaluating-third-party-llms-vertex-ai-gen-ai-eval-sdk

3 min read

Evaluating Third-Party LLMs with the Vertex AI Gen AI Evaluation SDK

Source notebook

Repo path: gemini/evaluation/evaluating_third_party_llms_vertex_ai_gen_ai_eval_sdk.ipynb · Open on GitHub · intermediate

Evaluates third-party, MaaS, BYOM, and Gemini models with Vertex AI Gen AI Evaluation.

Summary

This notebook teaches how to use the Vertex AI Gen AI Evaluation SDK through the vertexai Client interface. It runs inference for LiteLLM-backed third-party APIs, Vertex AI Model Garden MaaS models, and a custom BYOM endpoint, then evaluates outputs with rubric and text metrics. It also demonstrates comparing multiple model response datasets side by side on shared prompts and rubrics.

Key code patterns

Create eval client

from vertexai import Client, types
 
client = Client(project=PROJECT_ID, location=LOCATION)

Initializes the Vertex AI client used for inference, rubric generation, and evaluation.

Run third-party inference

openai_responses = client.evals.run_inference(
    model="gpt-5-mini",
    src="gs://vertex-evaluation-llm-dataset-us-central1/genai_eval_sdk/test_prompts.jsonl",
)

Uses a LiteLLM model string so the eval SDK can call a third-party provider using its API key.

Evaluate with mixed metrics

eval_result = client.evals.evaluate(
    dataset=openai_responses,
    metrics=[
        types.RubricMetric.GENERAL_QUALITY,
        types.RubricMetric.INSTRUCTION_FOLLOWING,
        types.Metric(name="rouge_1"),
        types.Metric(name="bleu"),
    ],
)

Combines rubric-based quality checks with automatic text similarity metrics.

Evaluate MaaS model

eval_dataset = client.evals.run_inference(
    model="deepseek-ai/deepseek-r1-0528-maas",
    src="gs://vertex-evaluation-llm-dataset-us-central1/genai_eval_sdk/test_prompts.jsonl",
)
maas_eval_result = client.evals.evaluate(dataset=eval_dataset, metrics=metrics)

Runs the same evaluation workflow against a managed Model Garden MaaS model.

Pass BYOM callable

def custom_model_inference_fn(prompt: str) -> str | None:
    response = requests.post(endpoint_url, headers=headers, data=json.dumps(payload))
    response.raise_for_status()
    return response.json()["predictions"]["choices"][0]["message"]["content"]
 
vertex_endpoint_responses = client.evals.run_inference(
    model=custom_model_inference_fn,
    src="gs://vertex-evaluation-llm-dataset-us-central1/genai_eval_sdk/test_prompts.jsonl",
)

Shows that run_inference can accept a Python function for independently served models.

Compare model datasets

comparison_eval_result = client.evals.evaluate(
    dataset=[gemini_dataset, deepseek_dataset, llama_dataset],
    metrics=[
        types.RubricMetric.GENERAL_QUALITY(rubric_group_name="general_quality_rubrics")
    ],
)

Evaluates multiple EvaluationDataset objects side by side with the same rubric metric.

Models & APIs used

  • Models: gpt-5-mini, deepseek-ai/deepseek-r1-0528-maas, meta/llama-3.1-70b-instruct-maas, gemini-2.5-flash, gemini-2.5-pro
  • APIs / services: Vertex AI, Vertex AI GenAI Evaluation Service, Vertex AI Model Garden, Cloud Storage
  • SDKs / libraries: google-cloud-aiplatform[evaluation], vertexai, litellm, google.colab, requests, pandas

When to use this

Use this pattern when you need one Vertex AI evaluation workflow for third-party APIs, Model Garden MaaS models, BYOM endpoints, and Gemini baselines.

Gotchas & caveats

  • Vertex AI API must be enabled and PROJECT_ID and LOCATION must be set.
  • The notebook uses billable Vertex AI components.
  • Third-party API evaluation requires provider API keys such as OPENAI_API_KEY or ANTHROPIC_API_KEY.
  • The notebook warns that setting API keys directly in code is insecure; use environment variables or secure storage.
  • The Colab userdata import is specific to Colab environments.
  • MaaS evaluation requires GOOGLE_APPLICATION_CREDENTIALS, VERTEXAI_PROJECT, VERTEXAI_LOCATION, and a service account with Vertex AI User role.
  • MaaS model availability depends on region.
  • BYOM inference depends on the dedicated endpoint URL, gcloud access token, and expected prediction response shape.
  • The 10-row comparison example is for demonstration, not rigorous benchmarking.

Best practices

  • Use environment variables or secure storage for provider API keys.
  • Enable Vertex AI API before running the workflow.
  • Set project and region from parameters or GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION.
  • Use the same dataset and metrics when comparing multiple models.
  • Generate rubrics from prompts before rubric-based comparison evaluation.
  • Use a Python callable to adapt custom endpoint inference into the eval SDK workflow.
  • Call show() on response and evaluation objects to inspect generated datasets and reports.

Graph View

Backlinks