The model as a component · Prompts and outputs

A schema for invoice line extraction

Lay out the output schema and the validation layer for invoice line extraction, deciding what the schema enforces, what code enforces, and what happens on failure.

designhardExercise · 4 min · 12 of 77

On this page

An extractor that returned valid JSON posted 120 wrong invoices in a month. The schema did what it promised; the design around it did not exist.

The situation

An accounts-payable team receives 2,000 supplier invoices a month as PDFs with the text already extracted, each with 1 to 40 line items. The first version asked the model to "return the line items as JSON" and posted the result straight to the ledger. In its first month, 6% of posted invoices — 120 — had line totals that did not sum to the invoice total, 2% carried a negative line, and 14 invoices were posted twice because the invoice number came back with a digit wrong. Reversing a bad posting takes an accountant about 20 minutes, so the month cost roughly (120 + 40 + 14) × 20 minutes ≈ 58 hours.

The ledger accepts three currencies and four tax rates. Suppliers occasionally send a credit note in the same inbox. The target is zero bad postings and a human review queue under three hours a month; a queued invoice with the failing check named takes about 3 minutes to review. The team is working from structured outputs and JSON Schema and has constrained decoding available for types, enums, required fields and additionalProperties.

The question

Design the output schema and the validation layer behind it: what the schema enforces, what code enforces, and what happens when a check fails.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

The schema, in decoding order. document_type as an enum of invoice and credit_note first, because it decides which invariants apply. Then currency and tax_rate, enums of the ledger's three currencies and four rates, so the model commits to the small choices before writing numbers. invoice_number and issue_date as strings, null allowed for a date the page does not show. line_items as an array of 1 to 60 objects: description (string, capped at 200 characters), quantity, unit_price and line_total as numbers. Then subtotal, tax and total as numbers, an unreadable boolean, and a free-text notes field last. Every field required, additionalProperties: false.

What code checks, in the invoice constructor: quantity × unit_price equals line_total within 0.01; the line totals sum to the subtotal; subtotal × tax_rate equals tax; subtotal + tax equals total; no negative quantity or price except on a credit note; the invoice number appears verbatim in the extracted text, so a wrong digit fails, and has not already been posted for this supplier, an idempotency key; and the currency matches the supplier record.

On failure: one retry with the failing check named — "line 3: 4 × 12.50 ≠ 60.00; subtotal off by 10.00" — then the review queue with that message attached, never a posting. unreadable: true goes to the queue directly.

The arithmetic: the invariants catch the 120 non-summing invoices and the 40 with a negative line, the verbatim and idempotency checks the 14 duplicates. Assume 2% still fail twice or are unreadable: 40 a month at 3 minutes each is two hours, under the three-hour target, against 58 hours today. Credit notes beyond a few a month get their own schema.

Rubric

orders the schema with enums first, numbers next and free text last, and says why decoding order matters
types amounts as numbers, admits null where a fact may be absent, and sets additionalProperties: false
separates what the schema enforces from what code enforces, with the arithmetic invariants listed
checks the invoice number against the source text, then uses it as an idempotency key against the ledger
defines the failure path: one named retry, then a review queue, never a posting
sizes the review queue against the target with the arithmetic shown

Where answers go wrong

"Use constrained decoding, and the output is valid." It is syntactically valid, which it already was: every one of the 120 bad invoices parsed. The reviewer is scoring whether the candidate knows the schema cannot add.

"Add 'make sure the line totals add up to the invoice total' to the prompt." A prompt line holds most of the time; the check in code holds every time, and it is the check that stops the posting. The line may reduce the failure rate, and it changes nothing about what reaches the ledger.

"Ask the model for a confidence score and post anything above 0.9." The score is a number the model generated, calibrated against nothing. An invoice whose lines do not sum can carry a confidence of 0.97, and did.