Generative AI Google Brain

gemini-controlled-generation-intro-controlled-generation

3 min read

Intro to Structured Output with the Gemini API

Source notebook

Repo path: gemini/controlled-generation/intro_controlled_generation.ipynb · Open on GitHub · intro

Shows how to make Gemini 2.5 Flash return JSON, enums, and schema-constrained multimodal outputs.

Summary

This notebook teaches controlled generation with the Gemini API in Vertex AI using the Google Gen AI SDK. It walks through JSON output with Pydantic models, OpenAPI-style schemas, and preview JSON Schema, then applies the same pattern to enums, game character data, log extraction, review sentiment, image object detection, and order updates with conditional fields.

Key code patterns

Create enterprise GenAI client

from google import genai
 
client = genai.Client(
    enterprise=True,
    project=PROJECT_ID,
    location=LOCATION,
)

Connects the notebook to a Google Cloud project and location before calling Gemini.

Pydantic JSON schema

class CountryInfo(BaseModel):
    name: str
    population: int
    capital: str
 
response = client.models.generate_content(
    model=MODEL_ID,
    contents="Fetch key facts for the United States.",
    config=GenerateContentConfig(
        response_mime_type="application/json",
        response_schema=CountryInfo,
    ),
)

Uses a Pydantic model as the response schema so Gemini returns predictable JSON.

Parsed structured response

country: CountryInfo = response.parsed
print(country.name)
print(country.population)
print(country.capital)

Shows that structured output can be consumed as a typed parsed object, not only raw text.

OpenAPI-style schema

response_schema = {
    "type": "OBJECT",
    "properties": {"name": {"type": "STRING"}},
    "required": ["name"],
}
 
config=GenerateContentConfig(
    response_mime_type="application/json",
    response_schema=response_schema,
)

Demonstrates dictionary schemas with required fields for JSON generation.

Preview JSON Schema

response = client.models.generate_content(
    model=MODEL_ID,
    contents="Fetch key facts for the United Kingdom.",
    config=GenerateContentConfig(
        response_mime_type="application/json",
        response_json_schema=country_info_schema,
    ),
)

Uses response_json_schema for JSON Schema features such as conditional requirements.

Enum output

class InstrumentEnum(Enum):
    PERCUSSION = "Percussion"
    STRING = "String"
 
response = client.models.generate_content(
    model=MODEL_ID,
    contents="What instrument plays multiple notes at once?",
    config=GenerateContentConfig(
        response_mime_type="text/x.enum",
        response_schema=InstrumentEnum,
    ),
)

Constrains the model to return one value from a predefined enum.

Multimodal structured output

contents=[
    Part.from_uri(
        file_uri="gs://cloud-samples-data/generative-ai/image/office-desk.jpeg",
        mime_type="image/jpeg",
    ),
    prompt,
]

Combines Cloud Storage image inputs with a JSON response schema for object detection.

Safety settings with schema

safety_settings=[
    SafetySetting(
        category="HARM_CATEGORY_DANGEROUS_CONTENT",
        threshold="BLOCK_LOW_AND_ABOVE",
    )
]

Shows safety controls can be passed alongside response_mime_type and response_schema.

Models & APIs used

  • Models: gemini-2.5-flash
  • APIs / services: Vertex AI, Agent Platform API, Cloud Storage
  • SDKs / libraries: google-genai, pydantic

When to use this

Use this pattern when an application needs Gemini responses in predictable JSON or enum formats for reliable downstream processing.

Gotchas & caveats

  • The notebook requires a Google Cloud project and the Agent Platform API enabled.
  • Colab users must run auth.authenticate_user before using the client.
  • PROJECT_ID falls back to GOOGLE_CLOUD_PROJECT and LOCATION falls back to GOOGLE_CLOUD_REGION.
  • The examples set LOCATION to global.
  • JSON Schema support is described as preview.
  • For OpenAPI schema, only enum, items, maxItems, nullable, properties, and required are listed as supported; other fields are ignored.
  • Fields are optional by default unless listed in required.
  • Cloud Storage image examples depend on gs://cloud-samples-data image URIs being accessible.

Best practices

  • Set response_mime_type to application/json for JSON outputs.
  • Use response_schema with Pydantic or OpenAPI-style schemas for controlled JSON generation.
  • Use response_json_schema when JSON Schema conditional logic is needed.
  • Use response.parsed to consume structured outputs as objects or dictionaries.
  • Use text/x.enum when the model must choose from predefined enum values.
  • Mark fields as required when the model must provide them.
  • Use nullable fields when the prompt may not contain enough context for a meaningful value.
  • Apply safety_settings together with controlled generation for content-sensitive prompts.

Graph View

Backlinks