Generative AI Google Brain

gemini-evaluation-evaltask-approach-prompt-engineering-gen-ai-evaluation-service-sdk

3 min read

Evaluate and Optimize Prompt Template Design for Better Results

Source notebook

Repo path: gemini/evaluation/evaltask_approach/prompt_engineering_gen_ai_evaluation_service_sdk.ipynb · Open on GitHub · intermediate

Compares Gemini prompt templates using Vertex AI EvalTask metrics to choose the best summarization prompt.

Summary

This notebook teaches how to use the Vertex AI Python SDK for Gen AI Evaluation Service to evaluate prompt template design. It builds a small summarization dataset with instruction, context, and reference fields, defines multiple prompt templates, runs each through Gemini 2.5 Flash with the same EvalTask metrics, and compares results with reports, explanations, radar plots, bar plots, and experiment runs.

Key code patterns

Initialize Vertex AI

PROJECT_ID = "[your-project-id]"
LOCATION = "us-central1"
 
if not PROJECT_ID or PROJECT_ID == "[your-project-id]":
    raise ValueError("Please set your PROJECT_ID")
 
import vertexai
vertexai.init(project=PROJECT_ID, location=LOCATION)

Sets the project and region before using Vertex AI evaluation and Gemini calls.

Generate from assembled prompt

client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)
 
model_response = client.models.generate_content(
    model="gemini-2.5-flash",
    contents=prompt_template,
).text

Shows direct Gemini inference through the Google GenAI SDK with Vertex AI enabled.

Create evaluation dataset

eval_dataset = pd.DataFrame({
    "context": context,
    "reference": reference,
    "instruction": [instruction] * len(context),
})

EvalTask uses structured columns for prompt variables and reference-based metrics.

Configure EvalTask

metrics = [
    "rouge_1", "rouge_l_sum", "bleu", "fluency", "coherence",
    "safety", "groundedness", "summarization_quality", "verbosity",
]
 
summarization_eval_task = EvalTask(
    dataset=eval_dataset,
    metrics=metrics,
    experiment=experiment_name,
)

Defines a reusable evaluation task with automatic, model-based, and text-overlap metrics.

Evaluate prompt variants

for i, prompt_template in enumerate(prompt_templates):
    experiment_run_name = f"eval-prompt-engineering-{run_id}-prompt-{i}"
    eval_result = summarization_eval_task.evaluate(
        prompt_template=prompt_template,
        experiment_run_name=experiment_run_name,
        model="gemini-2.5-flash",
    )
    eval_results.append((f"Prompt #{i}", eval_result))

Runs each prompt template under the same task, model, and metrics for comparison.

Display comparisons

for title, eval_result in eval_results:
    notebook_utils.display_eval_result(title=title, eval_result=eval_result)
    notebook_utils.display_explanations(eval_result, metrics=["summarization_quality"])
 
notebook_utils.display_radar_plot(eval_results, metrics=metrics)
notebook_utils.display_bar_plot(eval_results, metrics=metrics)

Surfaces aggregate scores and explanations to choose the strongest prompt template.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI, Vertex Gen AI Evaluation Service
  • SDKs / libraries: google-cloud-aiplatform[evaluation], google-genai, vertexai, pandas

When to use this

Use this pattern when you need to systematically compare prompt templates for a Gemini summarization task using Vertex AI evaluation metrics.

Gotchas & caveats

  • After installing google-cloud-aiplatform[evaluation] and google-genai, the notebook says the runtime must restart.
  • Colab requires auth.authenticate_user() before continuing.
  • PROJECT_ID must be set or the notebook raises ValueError.
  • The evaluation dataset must include instruction, context, and reference fields for pointwise inference.
  • The notebook says at least one evaluation example is required and recommends around 100 examples for higher-quality aggregate metrics.
  • Online evaluation quotas may affect performance and user experience.

Best practices

  • Use one shared EvalTask so prompt templates are compared against the same dataset, metrics, and experiment.
  • Use prompt template variables such as {instruction} and {context} to compile prompts from dataset fields.
  • Evaluate multiple prompt templates systematically before choosing one.
  • Use a consistent model, gemini-2.5-flash, across prompt variants for a fair comparison.
  • Inspect both evaluation results and summarization_quality explanations, not only aggregate scores.
  • Use radar and bar plots to compare prompt performance across all metrics.
  • Use generated run IDs and experiment run names to organize prompt comparison runs.

Graph View

Backlinks