Structured outputs and JSON Schema

When the model's output is read by code rather than by a person, the output is data, and data has a schema. Two mechanisms get you valid JSON — constrained decoding and parse-and-retry — and neither gets you correct JSON. The validation layer that checks totals and cross-field rules is still yours to write, and it is where money is lost when it is missing.

Two ways to get valid JSON

Parse-and-retry is the mechanism you can build on any API. Ask for JSON matching the schema, parse the reply, validate it, and on failure send the error back — "line 3: expected a number, got a string" — and ask again, capped at two attempts. Every retry is a full call with the prompt resent. At 10,000 calls a day and an 8% first-attempt failure rate, that is 800 retries of ~2k tokens: 1.6M extra input tokens a day, 1.6M × P_in, and a second call's latency for the 8% of users who hit it.

Constrained decoding moves the check into the sampler. The schema is compiled to a grammar, and at every step the tokens that would break the grammar are masked out before one is drawn. The output is syntactically valid by construction: no retry for syntax, no extra calls. The schema is still sent as input tokens, and the API's structured-outputs reference lists which JSON Schema keywords it enforces — typically types, enums, required fields and additionalProperties, often not patterns, numeric ranges or string lengths. Anything outside the enforced subset is a check you write.

Expand

A structured-output call: the schema shapes decoding, the schema check catches syntax, the invariant check catches arithmetic, and only a passing result reaches the ledger100%−+FitSVG (opens in a new tab)CloseEsc

Pinch to zoom · drag to panScroll to zoom · drag or arrow keys to pan · 0 fits · Esc closes

One property of decoding shapes the schema itself: the model writes the object left to right, so a field earlier in the object is decided before a later one, and a later field is generated conditioned on the earlier ones.

Writing the schema

A triage schema for the ticket classifier from prompt design:

{
  "type": "object",
  "properties": {
    "category": { "enum": ["billing", "bug", "how-to", "account", "other"] },
    "priority": { "enum": ["p1", "p2", "p3"] },
    "needs_human": { "type": "boolean" },
    "order_id": { "type": ["string", "null"] },
    "summary": { "type": "string", "maxLength": 280 }
  },
  "required": ["category", "priority", "needs_human", "order_id", "summary"],
  "additionalProperties": false
}

Enums first. The category is a one-token decision, and once decoded it conditions everything after it; a summary written before the label pulls the label toward the summary's framing. Free text last, for the same reason and for cost: the enums are one token each and the summary is ~70, so the structured fields are decided before the expensive band starts.

Numbers as numbers. An amount typed as a string arrives as "1,240.50" in one call and "1240.5" in the next, and the parser you write to normalise it is a bug farm. Null where the fact may be absent: a required field the model cannot fill is a field the model will invent, so order_id admits null and the absence is explicit rather than fabricated. Every field required, so that a missing key is a schema failure instead of a silent default.

additionalProperties: false. Without it the model adds "confidence": 0.97 — a number it made up, which your parser ignores and your reviewer believes.

The schema is input on every call. A 400-token schema at 10,000 calls a day is 4M tokens a day, 4M × P_in; keep it in the stable prefix so prompt caching reads it at C_r × P_in instead. A tool definition is the same object with a name and a description, and every rule here applies to it.

What a schema cannot say

Cross-field rules: if the category is billing, order_id must not be null. Totals: the line totals sum to the subtotal, and subtotal plus tax equals the total. Referential facts: the order exists, and it belongs to this customer. Temporal order: the due date is after the issue date. JSON Schema has if/then and dependentRequired for a few of these, and constrained decoders rarely enforce them; arithmetic, no schema says at all.

The mistake, with numbers. An accounts-payable team trusted "valid JSON" from an invoice extractor whose schema said line_total was a number. It was always a number. On 2% of invoices a line total was negative, and on 6% of invoices the lines did not sum to the total, and the ledger posted all of them because the schema check passed. 2,000 invoices a month × 6% = 120 postings that did not balance, each 20 minutes to reverse: 40 hours a month of an accountant's time, from a schema that did exactly what it promised. The invoice line extraction exercise designs the fix.

The validation layer you still write

Three checks, in order, each cheaper than a model call by orders of magnitude. The schema check: types, enums, required keys — microseconds, and already done when decoding is constrained. The invariant check: sums, ranges, dates, the enum-dependent requirements — code you write, and the place for it is the constructor of the object you build from the JSON, as constructors and invariants puts it: an object that cannot exist in a bad state. The referential check: the order id is in the database, the clause number is in the retrieved set — one lookup, a millisecond.

On failure: one retry with the failing check named in the message, because "line 3: 4 × 12.50 ≠ 60.00" is a repair instruction and "invalid output" is not. A second failure goes to a review queue with that message attached — never to a default value, and never to the ledger. These are the output gates of guardrails as gates, not filters; the stop-reason and provenance checks that sit beside them are in reading the response.

In an interview

What is being tested is whether you know where decoding's guarantee ends and your code begins. Say: "constrained decoding gives me syntax — types, enums, required, no extra keys. Cross-field rules, totals and references are invariants I check in the constructor, and a failure retries once with the error named, then goes to a person." Then name the order of fields and why: enums first, free text last, nulls where a fact may be absent.

The mistake that loses points is "we use structured outputs, so the output is validated" — the syntax is validated and the sum is not. The second is a schema with no additionalProperties: false and a confidence field that nobody calibrated.

Check yourself

1. An extractor on parse-and-retry returns invalid JSON 8% of the time, at 10,000 calls a day and 2k tokens each. Do you switch to constrained decoding?

The retries cost 800 × 2k = 1.6M input tokens a day, 1.6M × P_in, plus a second call's latency for 8% of users; constrained decoding removes both at no extra call. Switch — unless the schema leans on a keyword the API does not enforce, in which case constrain what it does enforce and keep that one check in code. The invariant layer stays either way.

2. A refund-decision schema has approve, amount and reason. In what order, and what does code check?

reason as an enum first — the category drives the rest — then approve as a boolean, then amount as a number, then any free-text note last. Code checks that approve implies amount greater than 0 and at most the order total, that the order exists, and that this order has not already been refunded; the last one is an idempotency check no schema can make.

3. The model keeps returning an extra confidence field that the product manager wants to show reviewers. Is that a problem?

With additionalProperties: false it cannot appear. Without it, the field is a number the model made up, and showing it is worse than hiding it. If the product wants a confidence, calibrate it: compare the field against 200 labelled items and report how often "0.9" is right before anyone acts on it — the method in sampling review and confidence thresholds.