Prompt design as interface design
Write a prompt as a five-part contract, test each revision against a labelled set, and read the confusion matrix to find the one instruction that moves the score.
On this page
A prompt is the request contract for an API whose implementation you cannot read. It has the parts a good endpoint has — who is calling, what is wanted, what is not allowed, what comes back, and examples of the exchange — and it earns trust the same way, by being tested against a labelled set rather than by reading well. The three revisions below moved a ticket classifier from 142 to 182 correct out of 200, and the change that did most of the work is not the one people expect.
Five parts, in this order
Role: one sentence saying what the reader is and for whom. Task: the verb and its object. Constraints: what is out of bounds, and the tie-break rules for the cases that sit between two answers. Output format: the exact shape of the reply, which is a schema whenever code reads it. Examples: two to five worked exchanges, chosen from the boundary cases, not the easy ones. Brown et al., "Language Models are Few-Shot Learners", showed that a model picks up a task from examples placed in the prompt alone, and that is still the cheapest instruction you can write for a case words describe badly.
The order is also a caching decision. The stable parts — role, task, constraints, format, examples — go first and stay byte-identical across calls; the per-request material, the ticket, goes last. Prompt caching matches a prefix, so a prompt that puts the ticket first forfeits the cache on every call.
The analogy is exact rather than decorative. In designing to an interface the caller depends on the contract and the implementation may change underneath; here the prompt is the contract the feature depends on, and the model is an implementation the provider changes without telling you. The contract is therefore the thing you test.
The worked example: a ticket classifier in three revisions
Setup. 200 labelled support tickets, five labels: billing 50, bug 45, how-to 45, account 35, other 25 — a golden set built in an afternoon. Each eval run is 200 calls of roughly 600 prompt tokens plus 150 of ticket, 5 tokens out: ~150k input tokens, 150k × P_in. At the course's latency figures (about 1 s to the first token and 50 tokens/s, from the four numbers) a call takes about 1.1 s, so with ten calls in flight a run is 20 rounds, about 20 seconds.
Revision 1, the one-liner. "You are a support assistant. Classify the following ticket as billing, bug, how-to, account or other." Score: 142 of 200, 71%. The matrix shows where the 58 errors live: bug and how-to are confused on 25 tickets, 12 one way and 13 the other; billing and account on 12; and 8 replies were sentences — "This looks like a billing issue" — which the parser mapped to other.
Revision 2, definitions and a format line. One sentence per label — "bug: the product did something other than documented; how-to: the product did what it does and the customer does not know how" — and "Reply with exactly one label from the list, lowercase, nothing else." Score: 166, 83%. The format line removed the sentence replies; the definitions cut bug-versus-how-to from 25 to 14.
Revision 3, the one instruction. "A ticket that mentions a charge, refund, invoice or card is billing, even if it also asks about logging in or a setting," plus two examples at the billing-account boundary and two at bug-how-to. Score: 182, 91%. Billing-account fell from 8 to 4 and bug-how-to from 14 to 6. A tie-break rule names the pair the model wavers between and says which side wins; the examples show the rule applied.
What did not move it. Between revisions 2 and 3 the role sentence was rewritten to "You are an expert support engineer with ten years of experience" as a control: 165, one ticket below revision 2, inside the noise of a 200-item set. The role sentence is a frame, not an instruction; the tie-break rule is an instruction.
What the matrix says that the score cannot
A score is one number; a confusion matrix is a map of where the errors are. The largest off-diagonal cell names a pair of labels, and a pair of labels is exactly what a tie-break rule is written for. The loop is: read the matrix, write the instruction for the biggest cell, re-run, read again. One change per revision, so every delta has a cause; a revision that changes four things and gains five points has taught you nothing about which of the four to keep.
Revision 3's 18 remaining errors: 6 bug-how-to, 4 billing-account, 5 between how-to and other (2 how-to tickets filed as other, 3 the other way), 3 scattered. The next revision is a sentence defining other more narrowly, and the expected gain is around 5 tickets. Below that, the remaining errors are the ambiguous tickets the labellers argued about, and the right move is to measure them with the 20-run repeat test rather than to keep writing rules.
Constraints that hold and constraints that do not
Positive beats negative: "reply with one label" works where "do not explain" does not, because the first describes the output and the second describes an absence. Specific beats general: a rule that names the pair beats "be careful with edge cases". Examples at the boundary beat examples of the obvious: an example the model would have classified correctly anyway spends tokens on every call and changes nothing.
Rules have a cost and a ceiling. Thirty rules at ~20 tokens each are 600 tokens on every call; at 10,000 calls a day that is 6M input tokens a day, 6M × P_in, before the ticket is read — and a rule sitting twentieth in a list is in the middle of the context, which the tokens lesson showed is where instructions are read worst. A prompt that has grown past a dozen rules is asking for a procedure to be moved out of it, which is what skills are for.
Some constraints do not belong in the prompt at all. "Never output a label outside the list" is a schema, and the decoder enforces it on every call. "Never delete a file" is a hook — code that runs before the tool call and rejects it, taught in hooks. A prompt line holds most of the time; a check in code holds always. The prompt carries judgment; the harness carries rules.
Versioning the contract
The prompt lives in the repository, with a version identifier that goes into every trace, so a change in behaviour on Tuesday can be matched to the revision that shipped on Monday. Every change to it runs the eval, the way every commit runs the test suite — the mechanism in evals as the unit test of LLM work. And because the model behind the name changes, the eval also runs on a model update with the prompt held fixed: a drop with no prompt change is the implementation moving under the contract, and the eval is how you find out before customers do.
In an interview
What is being tested is whether you treat prompting as engineering with a measurement attached. Say the five parts in order; say "200 labelled items, one change per revision, and the confusion matrix tells me which rule to write next"; and name a number — "revision 3 scored 182, and the role sentence moved nothing". Mention that the prompt is versioned and the eval runs on every change.
The mistake that loses points is listing prompt tricks — be polite, claim expertise, offer a tip — with no number attached to any of them, or saying a revision "improved it" without naming the set it improved on. The second is a prompt with thirty rules and no answer to "which of those is a schema, and which is a hook".
Check yourself
1. Your matrix shows 20 errors, 14 of them account tickets predicted as billing. What is the next revision, and what tells you it overshot?
One tie-break rule for the account-billing pair — "a ticket about a plan, a login or a profile is account unless it mentions a specific charge" — plus two boundary examples, and nothing else. Expect that cell to shrink. If the reverse cell, billing predicted as account, grows by more than it shrank, the rule overshot and the boundary needs an example on the other side.
2. The rules section has reached 3k tokens at 20,000 calls a day. Is caching the fix, and what has to be true for it to work?
3k × 20,000 = 60M input tokens a day on rules alone. With caching, the prefix is read at
C_r×P_ininstead ofP_in, so the daily saving is 60M × (1 −C_r) ×P_in— worth doing now. The prefix has to be byte-identical across calls, so the ticket goes last and nothing per-request — a timestamp, a user name — sits inside the rules. Then look at which of the 3k are procedures that belong in a skill rather than in every call.
3. A revision scores 185 against the previous 182. Do you ship it?
Not on that number. Three tickets of 200 is 1.5 points, and a pass rate near 90% on 200 items has a standard error of about 2 points, so the two revisions are inside noise. Look at the paired diff — which tickets flipped, and which way — and re-run each three times before deciding; regression gates is the arithmetic.