Generative AI Google Brain

gemini-logprobs-intro-logprobs

3 min read

Intro to Logprobs

Source notebook

Repo path: gemini/logprobs/intro_logprobs.ipynb · Open on GitHub · intermediate

Introduces Gemini logprobs on Vertex AI for confidence, autocomplete, and RAG grounding analysis.

Summary

This notebook teaches how to enable response_logprobs and logprobs in Gemini API calls on Vertex AI using the Google Gen AI SDK. It demonstrates inspecting chosen and alternative token log probabilities, then applies them to classification ambiguity detection, confidence thresholding, autocomplete suggestions, and a simple RAG grounding score workflow.

Key code patterns

Create Vertex AI GenAI client

from google import genai
 
client = genai.Client(
    vertexai=True,
    project=PROJECT_ID,
    location=LOCATION,
)

Authenticates Google Gen AI SDK calls against Vertex AI with a project and location.

Enable response logprobs

response = client.models.generate_content(
    model=MODEL_ID,
    contents=prompt,
    config=GenerateContentConfig(
        response_logprobs=True,
        logprobs=3,
    ),
)

Requests log probabilities for generated tokens plus top alternative token candidates.

Structured enum classification

response_schema = {
    "type": "STRING",
    "enum": ["Positive", "Negative", "Neutral"],
}
config = GenerateContentConfig(
    response_mime_type="text/x.enum",
    response_schema=response_schema,
    response_logprobs=True,
    logprobs=3,
)

Constrains output to labels so logprobs can be compared across classification choices.

Convert logprob to confidence

chosen = response.candidates[0].logprobs_result.chosen_candidates[0]
probability = math.exp(chosen.log_probability)
if probability >= threshold:
    return chosen.token
return None

Turns the chosen token log probability into a probability score for thresholding.

Average logprob RAG score

total_logprob = 0.0
token_count = 0
for candidate in logprobs_result.chosen_candidates:
    total_logprob += candidate.log_probability
    token_count += 1
average_logprob = total_logprob / token_count

Uses average chosen-token logprob as a simple confidence or grounding score.

Models & APIs used

  • Models: gemini-3.5-flash
  • APIs / services: Vertex AI, Gemini API
  • SDKs / libraries: google-genai, pandas

When to use this

Use this pattern when you need token-level confidence signals from Gemini outputs for classification, autocomplete, or RAG answer evaluation.

Gotchas & caveats

  • Requires Google Cloud credentials and a project ID for Vertex AI access.
  • Colab users must run auth.authenticate_user() before using Google Cloud credentials.
  • LOCATION defaults to global from GOOGLE_CLOUD_REGION if unset.
  • logprobs is documented in the notebook as an integer range from 1 to 20.
  • The notebook notes outputs may differ between runs.
  • RAG evaluation uses a fictional knowledge base and a simple average logprob heuristic, not a full retrieval system.

Best practices

  • Use response_logprobs=True only when token confidence data is needed.
  • Use logprobs to inspect top alternative tokens and debug model behavior.
  • Constrain classification outputs with response_schema and text/x.enum before comparing label confidence.
  • Flag classifications for human review when top choices have close log probabilities.
  • Convert log probabilities with math.exp before applying probability thresholds.
  • Use temperature=0 for deterministic factual RAG evaluation examples.
  • Prompt RAG answers to respond based only on provided context.

Graph View

Backlinks