Generative AI Google Brain

gemini-evaluation-evaltask-approach-compare-and-migrate-from-palm-to-gemini

3 min read

Migrate from PaLM to Gemini model

Source notebook

Repo path: gemini/evaluation/evaltask_approach/compare_and_migrate_from_palm_to_gemini.ipynb · Open on GitHub · intermediate

Compares PaLM text-bison and gemini-2.5-flash with Vertex AI EvalTask for summarization migration.

Summary

The notebook teaches how to use Vertex AI Gen AI Evaluation Service EvalTask to compare PaLM text-bison@001 against gemini-2.5-flash for summarization prompts with references. It installs and initializes the Vertex AI SDK, builds a pandas evaluation dataset, evaluates both models across multiple metrics, and visualizes results with radar and bar plots. It concludes that gemini-2.5-flash outperformed text-bison for this specific use case and recommends migrating for that task.

Key code patterns

Initialize Vertex AI

import vertexai
 
PROJECT_ID = "[your-project-id]"
LOCATION = "us-central1"
if not PROJECT_ID or PROJECT_ID == "[your-project-id]":
    raise ValueError("Please set your PROJECT_ID")
vertexai.init(project=PROJECT_ID, location=LOCATION)

Sets the project and location before running Vertex AI evaluation calls.

Build Evaluation Dataset

eval_dataset = pd.DataFrame({
    "prompt": [instruction + item for item in context],
    "reference": reference,
})

Creates the prompt and reference columns EvalTask uses for summarization evaluation.

Define EvalTask Metrics

metrics = [
    "rouge_l_sum", "bleu", "fluency", "coherence",
    "safety", "groundedness", "verbosity",
    "text_quality", "summarization_quality",
]
eval_task = EvalTask(
    dataset=eval_dataset,
    metrics=metrics,
    experiment=experiment_name,
)

Combines automatic and model-based metrics in one evaluation task.

Wrap PaLM Predictor

generation_config = {"temperature": 0.5, "max_output_tokens": 256, "top_k": 1}
text_bison_model = TextGenerationModel.from_pretrained("text-bison@001")
 
def text_bison_model_fn(prompt):
    return text_bison_model.predict(prompt, **generation_config).text

Uses a callable so EvalTask can evaluate the PaLM TextGenerationModel output.

Evaluate Gemini Model ID

gemini_eval_result = eval_task.evaluate(
    model="gemini-2.5-flash",
    experiment_run_name=experiment_run_name,
    evaluation_service_qps=5,
)

Shows EvalTask evaluating Gemini directly from its model ID string.

Visualize Model Comparison

results = [
    ("text-bison", text_bison_eval_result),
    ("gemini-2.5-flash", gemini_eval_result),
]
notebook_utils.display_radar_plot(results, metrics=["fluency", "coherence", "safety", "groundedness", "verbosity", "text_quality", "summarization_quality"])
notebook_utils.display_bar_plot(results, metrics=["rouge_l_sum", "bleu"])

Uses radar plots for qualitative metrics and bar plots for ROUGE/BLEU.

Models & APIs used

  • Models: text-bison@001, gemini-2.5-flash
  • APIs / services: Vertex AI, Vertex Gen AI Evaluation Service
  • SDKs / libraries: google-cloud-aiplatform[evaluation], vertexai, pandas

When to use this

Use this pattern when deciding whether to migrate a summarization workflow from PaLM text-bison to gemini-2.5-flash using evaluation metrics.

Gotchas & caveats

  • Colab users must restart the runtime after installing google-cloud-aiplatform[evaluation].
  • Colab authentication is required before using the notebook environment.
  • PROJECT_ID must be set and the Vertex AI API must be enabled before vertexai.init.
  • LOCATION is configured as us-central1 in the notebook.
  • The notebook recommends at least 100 examples for best results, but the sample dataset has five rows.
  • Both evaluate calls set evaluation_service_qps=5, limiting evaluation service request rate.

Best practices

  • Use an EvalTask with multiple metrics when comparing foundation models.
  • Include prompt and reference columns in the summarization evaluation dataset.
  • Generate unique experiment run names with notebook_utils.generate_uuid(8).
  • Use at least 100 examples for best evaluation results.
  • Visualize qualitative metrics separately from ROUGE and BLEU.
  • Compare models on the target use case before recommending migration.

Graph View

Backlinks