Generative AI Google Brain

open-models-evaluation-vertex-ai-tgi-evaluate-llm-with-open-judge

3 min read

Using open autorater for running evaluations with Vertex AI Gen AI Evaluation

Source notebook

Repo path: open-models/evaluation/vertex_ai_tgi_evaluate_llm_with_open_judge.ipynb · Open on GitHub · advanced

Deploys Selene as an open judge on Vertex AI and uses Gen AI Evaluation to score LLM responses.

Summary

This notebook teaches how to use an open-source autorater model deployed to a Vertex AI Endpoint for pointwise LLM evaluation. It uploads and deploys AtlaAI/Selene-1-Mini-Llama-3.1-8B with a Hugging Face TGI container, prepares a human-rated dataset, defines a custom completeness metric, runs EvalTask with AutoraterConfig, and compares autorater scores to human ratings with evaluate_autorater and Plotly visualizations.

Key code patterns

Initialize Vertex AI

vertexai.init(
    project=PROJECT_ID,
    location=LOCATION,
    staging_bucket=BUCKET_URI,
    experiment=EXPERIMENT_NAME,
)

Sets project, region, staging bucket, and experiment tracking before model deployment and evaluation.

Upload Open Judge

judge_model = aiplatform.Model.upload(
    display_name="google--selene-1-mini-llama-3.1-8b",
    serving_container_image_uri="us-docker.pkg.dev/deeplearning-platform-release/gcr.io/huggingface-text-generation-inference-cu124.2-4.ubuntu2204.py311",
    serving_container_environment_variables={
        "MODEL_ID": "AtlaAI/Selene-1-Mini-Llama-3.1-8B",
        "NUM_SHARD": "1",
        "HUGGING_FACE_HUB_TOKEN": get_token(),
    },
    serving_container_ports=[8080],
)

Registers the Hugging Face Selene model in Vertex AI using a prebuilt Text Generation Inference container.

Deploy Judge Endpoint

deployed_judge_model = judge_model.deploy(
    endpoint=aiplatform.Endpoint.create(
        display_name="google--selene-1-mini-llama-3.1-8b-endpoint"
    ),
    machine_type="g2-standard-4",
    accelerator_type="NVIDIA_L4",
    accelerator_count=1,
)

Makes the judge model callable through a Vertex AI Endpoint with specified GPU hardware.

Create Pointwise Metric

completeness_prompt_template_fmt = tokenizer.apply_chat_template(
    completeness_messages,
    tokenize=False,
    add_generation_prompt=True,
)
 
completeness = PointwiseMetric(
    metric="completeness",
    metric_prompt_template=completeness_prompt_template_fmt,
)

Wraps a rubric-based prompt template into a Vertex AI Evaluation metric.

Run Evaluation

eval_result = EvalTask(
    dataset=human_rated_dataset,
    metrics=[completeness],
    experiment=EXPERIMENT_NAME,
    autorater_config=AutoraterConfig(
        autorater_model=deployed_judge_model.resource_name
    ),
    output_uri_prefix=BUCKET_URI + "/evaluation_results",
).evaluate()

Runs the Gen AI Evaluation task using the deployed Selene endpoint as the autorater.

Meta-Evaluate Autorater

evaluate_autorater_result = evaluate_autorater(
    evaluate_autorater_input=eval_result.metrics_table,
    eval_metrics=[completeness]
)
 
df_data = eval_result.metrics_table[[
    "completeness/human_rating",
    "completeness/score",
]]

Compares autorater scores with human ratings to assess alignment.

Models & APIs used

  • Models: AtlaAI/Selene-1-Mini-Llama-3.1-8B
  • APIs / services: Vertex AI, Vertex AI Gen AI Evaluation, Cloud Storage
  • SDKs / libraries: google-cloud-aiplatform, vertexai, huggingface_hub, transformers, pandas, numpy, plotly

When to use this

Use this pattern when you need a custom rubric-based LLM judge hosted on Vertex AI and want to compare its scores against human ratings.

Gotchas & caveats

  • Vertex AI API must be enabled for the Google Cloud project.
  • A Cloud Storage bucket is created and used as the Vertex AI staging bucket and evaluation output prefix.
  • Hugging Face authentication is required to download the Selene model artifacts and tokenizer.
  • The deployment specifies us-central1 by default unless GOOGLE_CLOUD_REGION is set.
  • The judge endpoint uses g2-standard-4 with one NVIDIA_L4 accelerator, so GPU quota and regional availability matter.
  • Cleanup is disabled by default because delete_endpoint and delete_experiments are set to False.

Best practices

  • Test the deployed judge endpoint with a sample prediction before running evaluation.
  • Use a structured evaluation dataset with user_input, ground_truth, assistant_response, and human rating columns.
  • Define the scoring rubric and required autorater output format explicitly in the metric prompt.
  • Use tokenizer.apply_chat_template for prompts sent to the Selene model.
  • Compare autorater scores against human ratings with evaluate_autorater before trusting judge alignment.
  • Visualize score distributions, confusion matrix, and item-level agreement.

Graph View

Backlinks