Generative AI Google Brain

search-gemini-enterprise-gemini-enterprise-eval

3 min read

Gemini Enterprise answer eval using BLEU, ROUGE, BERT, Similarity Score

Source notebook

Repo path: search/gemini-enterprise/gemini_enterprise_eval.ipynb · Open on GitHub · intermediate

Evaluates Gemini Enterprise answers against a golden dataset with NLP metrics and saves results.

Summary

This notebook builds a repeatable evaluation flow for Gemini Enterprise answer quality. It authenticates to Google Cloud, loads golden queries from CSV or Google Sheets, retrieves generated answers through Discovery Engine Answer APIs, scores them with BLEU, ROUGE-L, BERTScore, and semantic similarity, then writes results to CSV, Google Sheets, and BigQuery.

Key code patterns

Location-aware Discovery Engine endpoint

if location == "us":
    base_discovery_engine_domain = "us-discoveryengine.googleapis.com"
elif location == "eu":
    base_discovery_engine_domain = "eu-discoveryengine.googleapis.com"
else:
    base_discovery_engine_domain = "discoveryengine.googleapis.com"

Keeps Discovery Engine REST calls aligned with the configured Gemini Enterprise location.

Answer API retrieval

session = requests.post(session_url, headers=headers, json={"userPseudoId": "12345"})
response = requests.post(answer_url, headers=headers, json={
    "query": {"text": query},
    "searchSpec": {"searchParams": {"maxReturnResults": K}},
    "session": session.json()["name"],
})
answer_data = response.json().get("answer").get("answerText")

Fetches Gemini Enterprise generated answers for each golden query before scoring.

Golden dataset from Google Sheets

drive.mount("/content/drive")
auth.authenticate_user()
creds, _ = default()
gc = gspread.authorize(creds)
spreadsheet = gc.open_by_url(eval_data_google_drive_url)
worksheet = spreadsheet.worksheet(worksheet_name)
df = pd.DataFrame(worksheet.get_all_values()[1:], columns=worksheet.get_all_values()[0])

Loads test queries and expected answers from a shared spreadsheet workflow.

Metric computation

reference = [nltk.word_tokenize(expected.lower())]
candidate = nltk.word_tokenize(actual.lower())
bleu = sentence_bleu(reference, candidate)
rouge = rouge_scorer.RougeScorer(["rougeL"], use_stemmer=True).score(expected, actual)["rougeL"].fmeasure
P, R, F1 = bert_score([actual], [expected], lang="en", verbose=False)
bert = F1[0].item()

Combines lexical overlap and contextual similarity for answer evaluation.

Append results to BigQuery

client = bigquery.Client(project=project_id)
table_id = f"{project_id}.{dataset_id}.{table_name}"
job_config = bigquery.LoadJobConfig(write_disposition="WRITE_APPEND")
job = client.load_table_from_dataframe(output_df, table_id, job_config=job_config)
job.result()

Stores timestamped evaluation runs for historical analysis.

Models & APIs used

  • Models: all-MiniLM-L6-v2
  • APIs / services: Discovery Engine, Gemini Enterprise Search, Gemini Enterprise Assist, Google Sheets API, Google Drive API, BigQuery, Vertex AI
  • SDKs / libraries: vertexai, google-auth, google-colab, gspread, google-cloud-bigquery, pandas, requests, nltk, rouge-score, bert-score, sentence-transformers

When to use this

Use this pattern when you need quantitative regression tracking for Gemini Enterprise Search or Assist answer quality against a golden dataset.

Gotchas & caveats

  • The code expects search_query and expected_answers columns, while the overview also mentions expected answers.
  • The main population step calls get_answer_results(q) even though app_type and get_assist_results are defined.
  • Vertex AI initialization uses us-central1 with a note that global is not supported yet.
  • Discovery Engine REST calls require refreshed credentials and Bearer token headers.
  • The notebook requires Discovery Engine, Sheets, Drive, and BigQuery APIs to be enabled.
  • NLTK resources punkt and punkt_tab are downloaded before tokenized scoring.

Best practices

  • Use a golden dataset with explicit query and expected-answer columns.
  • Evaluate each answer with multiple metrics instead of a single score.
  • Add timestamps to evaluation outputs for run tracking.
  • Convert tuple ratings and timestamps to strings before writing to Sheets or BigQuery.
  • Create BigQuery datasets and tables if they do not already exist.
  • Clear existing logger handlers before adding notebook log handlers.

Graph View

Backlinks