Generative AI Google Brain

gemini-use-cases-entity-extraction-evaluate

2 min read

Evaluate

Source notebook

Repo path: gemini/use-cases/entity-extraction/evaluate.ipynb · Open on GitHub · intermediate

Evaluates Gemini-based document classification on image samples and visualizes confusion matrices.

Summary

This notebook loads Gemini project settings from a .env file, sets gemini-2.5-flash as the evaluation model, and runs document classification evaluation through local document_processing and evaluate modules. It first evaluates a stratified 120-sample set across all classes, then evaluates a 60-sample subset for budget, specification, form, and invoice. The workflow prints summary metrics, plots ordered confusion matrices, and displays the selected-class evaluation DataFrame.

Diagrams

current_architecture.png — source

future_architecture.png — source

Key code patterns

Environment-driven configuration

dotenv.load_dotenv(dotenv_path=".env", override=True)
PROJECT_ID = os.environ.get("GEMINI_PROJECT_ID")
if not PROJECT_ID:
    raise ValueError("GEMINI_PROJECT_ID environment variable must be set.")
LOCATION = os.environ.get("GEMINI_LOCATION", "global")
IMAGE_PATHS = os.environ.get("IMAGE_PATHS", "")
IMAGE_PREFIX = os.environ.get("IMAGE_PREFIX", "")

Keeps project, location, and image inputs outside notebook code.

Stratified all-class evaluation

result, df = evaluate.run_evaluation(
    project_id=PROJECT_ID,
    location=LOCATION,
    csv_path=IMAGE_PATHS,
    image_prefix=IMAGE_PREFIX,
    eval_model="gemini-2.5-flash",
    sample_size=120,
    random_state=42,
    stratify=True,
)

Runs reproducible evaluation over a stratified sample.

Focused class evaluation

target_classes = ["budget", "specification", "form", "invoice"]
selected_result, selected_df = evaluate.run_evaluation(
    project_id=PROJECT_ID,
    location=LOCATION,
    csv_path=IMAGE_PATHS,
    image_prefix=IMAGE_PREFIX,
    eval_model=EVAL_MODEL,
    sample_size=60,
    classes=target_classes,
)

Narrows evaluation to classes of particular interest.

Ordered confusion matrix

avg_df = df.groupby("reference")["exact_match"].mean().reset_index()
ordered_classes = avg_df.sort_values("exact_match")["reference"].tolist()
cm = pd.crosstab(df["reference"], df["response"])
sns.heatmap(cm.reindex(index=full_order, columns=full_order, fill_value=0), annot=True, fmt="d")

Orders labels by exact-match performance to make weak classes easier to inspect.

Models & APIs used

  • Models: gemini-2.5-flash
  • SDKs / libraries: dotenv, pandas, matplotlib, seaborn

When to use this

Use this pattern to evaluate Gemini document classification quality on labeled image datasets and inspect errors by class.

Gotchas & caveats

  • GEMINI_PROJECT_ID must be set or the notebook raises ValueError.
  • GEMINI_LOCATION defaults to global when not set.
  • IMAGE_PATHS and IMAGE_PREFIX are read from environment variables and default to empty strings.
  • The notebook depends on local document_processing and evaluate modules.
  • Evaluation sample sizes are fixed in the notebook at 120 and 60.

Best practices

  • Use environment variables for project, location, and image path configuration.
  • Set RANDOM_STATE for reproducible sampling.
  • Use stratify=True for class-balanced evaluation samples.
  • Evaluate both all classes and a targeted subset of high-interest classes.
  • Inspect summary metrics together with confusion matrices and detailed rows.

Graph View

Backlinks