OpenAI Structured Outputs API feature announcement

TL;DR

OpenAI announced Structured Outputs for its API, a feature that guarantees model responses conform exactly to developer‑provided JSON Schemas, improving reliability for data‑centric applications.

What Structured Outputs Are

Structured Outputs enforce that the model’s output matches a JSON Schema supplied by the developer. This goes beyond the earlier JSON mode, which only encouraged valid JSON but did not ensure adherence to a specific schema.

How to Use Structured Outputs

Function‑calling interface

  • Set strict: true in a tool definition.
  • Works with any model that supports tools (e.g., gpt‑4‑0613, gpt‑3.5‑turbo‑0613, and newer models).
  • Example request shows a query function with a detailed schema for table name, columns, conditions, and ordering. The model returns a JSON object that exactly matches this schema.

response_format interface

  • Provide a JSON Schema via the new json_schema option inside response_format.
  • Available on the newest GPT‑4o models: gpt‑4o‑2024‑08‑06 and gpt‑4o‑mini‑2024‑07‑18.
  • Example request formats a math‑tutoring response with a steps array and a final_answer field, and the model returns data that conforms to the schema.

Safety Guarantees

  • Structured Outputs respect existing safety policies; the model can still refuse unsafe requests.
  • When a refusal occurs, the API includes a refusal field in the response, allowing developers to detect non‑conforming output programmatically.

Native SDK Support

  • Updated Python and Node SDKs accept Pydantic (Python) or Zod (Node) objects directly.
  • The SDK converts these typed objects to JSON Schemas, sends them to the API, and deserializes the returned JSON back into the original typed structures.
  • Example code demonstrates parsing a Query Pydantic model and a MathResponse model.

Key Use Cases

  • Dynamic UI generation – Generate UI component trees that match a recursive schema, enabling on‑the‑fly interface creation.
  • Separating reasoning from final answer – Return a reasoning_steps array and a concise answer field, improving transparency.
  • Extracting structured data – Pull action items, due dates, and owners from free‑form meeting notes into a well‑defined schema.

Technical Under the Hood

Constrained decoding

  • The model’s token sampler is limited at each step to tokens that keep the output valid under the supplied schema.
  • The JSON Schema is compiled into a context‑free grammar (CFG). During generation, the inference engine masks out invalid tokens based on the current partial output.
  • The first request with a new schema incurs preprocessing latency (typically under 10 seconds; up to a minute for complex schemas) to build the grammar cache.

Why CFG over FSM/regex

  • CFGs can represent recursive structures, which FSMs cannot handle reliably.
  • This enables support for schemas with nested or self‑referencing objects, such as dynamically generated UI component trees.

Limitations

  • Only a subset of JSON Schema is supported (see the docs for the exact list).
  • First‑use latency for a new schema; subsequent calls are fast.
  • Refusals, token limits, or early stop reasons can cause the model to return a non‑conforming response.
  • Values inside the JSON may still be incorrect; developers should provide examples or break tasks into smaller subtasks.
  • Parallel tool calls are not compatible; set parallel_tool_calls: false to avoid mismatches.
  • Schemas used with Structured Outputs are not eligible for Zero Data Retention.

Availability and Pricing

  • Structured Outputs are generally available today across the Chat Completions, Assistants, and Batch APIs.
  • Function‑calling mode works with all models that support tools, including gpt‑4o, gpt‑4o‑mini, and any fine‑tuned models with tool support.
  • response_format mode works with gpt‑4o‑2024‑08‑06, gpt‑4o‑mini‑2024‑07‑18, and compatible fine‑tunes.
  • Switching to gpt‑4o‑2024‑08‑06 reduces input costs by 50 % and output costs by 33 % compared with the May 2024 version.

Acknowledgements

OpenAI credits the open‑source community for inspiration, naming projects such as outlines, jsonformer, instructor, guidance, and lark as influences on the Structured Outputs implementation.

This article summarizes OpenAI’s official announcement of Structured Outputs released on August 6 2024.

Sources