Generative AI Google Brain

gemini-evaluation-evaluate-gemini-structured-output

3 min read

Evaluate Gemini Structured Output

Source notebook

Repo path: gemini/evaluation/evaluate_gemini_structured_output.ipynb · Open on GitHub · intermediate

Evaluates Gemini structured JSON extraction from scanned order forms with Vertex AI Gen AI Evaluation.

Summary

This notebook teaches how to compare Gemini models on a scanned handwritten order-form extraction task. It configures Gemini structured output with a JSON schema, runs multiple models over Cloud Storage images, builds a ground-truth evaluation dataset, and evaluates results with Vertex AI Gen AI Evaluation using exact match plus custom schema and accuracy metrics. It also uses Gemini 2.5 Flash to summarize the experiment results.

Key code patterns

Vertex AI setup

client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)
vertexai.init(project=PROJECT_ID, location=LOCATION)

Uses the Google GenAI SDK against Vertex AI and initializes Vertex AI evaluation context.

Structured JSON config

generate_content_config = GenerateContentConfig(
    response_mime_type="application/json",
    response_schema=schema,
)

Constrains Gemini responses to JSON shaped by the order-form schema.

Image extraction loop

image = Part.from_uri(file_uri=image_info["image_uri"], mime_type=image_info["image_type"])
response = client.models.generate_content(
    model=model,
    contents=[prompt, image],
    config=generate_content_config,
)
response_json = json.dumps(response.parsed, indent=4)

Runs each Gemini model on each scanned form and stores parsed structured output.

Evaluation rows

eval_dataset_rows.append({
    "model": model_name,
    "prompt": prompt,
    "image": image_name,
    "reference": reference_str,
    "response": response_text,
    "differences": DeepDiff(json.loads(reference_str), json.loads(response_text)).pretty(),
})
eval_dataset = pd.DataFrame(eval_dataset_rows)

Creates a bring-your-own-response dataset with references, responses, and field-wise differences.

Schema metric

def is_valid_schema(instance):
    try:
        validate(instance=json.loads(instance["response"]), schema=schema)
    except Exception:
        return {"valid_schema": False}
    return {"valid_schema": True}
valid_schema = CustomMetric(name="valid_schema", metric_function=is_valid_schema)

Adds a custom metric that verifies whether generated JSON conforms to the schema.

Accuracy metric

def calculate_accuracy(instance):
    try:
        reference_data = json.loads(instance["reference"])
        response_data = json.loads(instance["response"])
    except json.JSONDecodeError:
        return {"accuracy": 0.0}
    deep_distance = DeepDiff(reference_data, response_data, ignore_order=True, get_deep_distance=True).get("deep_distance")
    return {"accuracy": 1.0 - deep_distance}

Computes a graded accuracy score from DeepDiff instead of only exact string equality.

EvalTask run

extraction_eval_task = EvalTask(
    dataset=eval_dataset,
    metrics=["exact_match", valid_schema, accuracy],
    experiment=EXPERIMENT_NAME,
)
eval_result = extraction_eval_task.evaluate(experiment_run_name=experiment_run_name)

Runs the Vertex AI evaluation experiment with built-in and custom metrics.

Models & APIs used

  • Models: gemini-2.0-flash, gemini-2.5-flash, gemini-2.5-pro
  • APIs / services: Vertex AI, Gen AI Evaluation Service, Cloud Storage
  • SDKs / libraries: google-genai, google-cloud-aiplatform, vertexai, jsonschema, deepdiff, pandas, IPython

When to use this

Use this pattern when comparing Gemini models for structured JSON extraction from images against ground-truth records.

Gotchas & caveats

  • Requires an existing Google Cloud project with the Vertex AI API enabled.
  • Colab requires auth.authenticate_user before running Google Cloud calls.
  • The notebook says to restart the runtime after installing packages.
  • LOCATION defaults to us-central1 and model access must match the project and location.
  • Cloud Storage image URIs must be accessible to the Vertex AI Gemini request.
  • The prompt asks for null on missing fields, but the schema uses string, integer, and number types without null.
  • exact_match is sensitive to JSON formatting, so the notebook treats custom accuracy as the better metric.

Best practices

  • Use structured output with response_mime_type application/json and response_schema for consistent JSON.
  • Keep reference ground truth alongside each model response in the evaluation dataset.
  • Evaluate both schema validity and extraction accuracy with custom metrics.
  • Use DeepDiff to inspect field-wise differences between reference and response.
  • Compare multiple Gemini model ids over the same prompt and input images.
  • Use an experiment name and generated run ids for repeatable evaluation tracking.

Graph View

Backlinks