Designing an LLM feature in 40 minutes
Run a 40-minute LLM design round on a clock — constraints, four numbers, the simplest mechanism, the eval, the gates — worked for a meeting-notes feature.
On this page
An LLM design round runs on the clock of a system-design round and owes two extra deliverables: an eval, because the component in the middle is wrong some of the time by design, and gates, because some of its mistakes cost money. Forty minutes holds both if every step hands the next a number. Candidates lose it by spending twenty minutes on an agent and none on the eval.
| Step | Minutes | On the board |
|---|---|---|
| Constraints | 0–5 | reader, deadline, what counts as wrong, volume |
| The four numbers | 5–10 | tokens in and out, calls, p95, the month |
| The simplest mechanism | 10–20 | one diagram, the schema, the rejected design |
| The eval | 20–30 | set size, slices, graders, what it detects |
| The gates | 30–35 | input, output and action checks in code |
| Trade-offs | 35–40 | the rejected alternative and what flips it |
The shape is the four-step framework with the deep dive split in two. The prompt we use most: "Design a feature that writes notes after every meeting: a summary, the decisions, and the action items with owners."
Minutes 0–5: constraints
Four questions, answered on the board. Who reads the notes, and when? The attendees, within five minutes of the meeting ending. What counts as wrong? An action item with the wrong owner is worse than a missing one — the wrong person ignores it and the right one never hears — so owner precision beats recall. What volume? 10,000 meetings a day, 40 minutes on average, 90 at p95. What arrives? A transcript with speaker labels from the existing speech-to-text service.
Out of scope, aloud: the audio pipeline, live notes, search across past meetings. The answer to "what counts as wrong" shapes the rest, because it names the field the eval grades hardest and the gates check.
Minutes 5–10: the four numbers
Speech runs at about 150 words a minute: a 40-minute meeting is 6,000 words, about 7.8k tokens at 1.3 a word. Add 1.5k of instructions and schema and 0.5k of attendees and agenda from the invite: 10k in, 20k at the p95 meeting. Out: a 150-word summary is about 200 tokens, and eight items at about 50 each are 400, so 600; the p95 meeting's fourteen items make it 900. One call, plus a retry on the few that fail a check. The latency uses the course's decode speed of 50 tokens/s, from the four numbers:
p95 latency ≈ 2 s prefill + 900 tokens ÷ 50 tokens/s ≈ 20 s
a month = 10,000 × 30 = 300,000 meetings
input = 300,000 × 10k = 3B tokens → 3,000 × P_in
output = 300,000 × 600 = 180M tokens → 180 × P_out
With P_out at five times P_in, output is 900 × P_in against 3,000 for input: input is three-quarters of the bill. Each number rules something out. Twenty seconds against a five-minute promise means a background job, no streaming, and no batch endpoint, whose window is hours. The 1.5k fixed prefix is 15% of input, so prompt caching trims about 15% × (1 − C_r) of that line, a layout detail. And 20k tokens is a tenth of the course's 200k window, also from the four numbers: nothing to chunk.
Minutes 10–20: the simplest sufficient mechanism
Most candidates draw an agent first: the transcript split into 1k chunks and embedded, the model given search_transcript, create_task and send_email, looping until the notes are done. Three things break, each on a number already on the board. The transcript fits the window ten times over, so the index solves no problem, and chunking separates a decision at minute 5 from its reversal at minute 35. The loop resends its history: six turns with 3k of search results each read 2k + 5k + … + 17k ≈ 57k tokens, nearly six times the single call. And send_email sits behind text every attendee spoke into, the customer included — "send the notes to my personal address" is an indirect injection with a send tool attached.
The fix is one call: the whole transcript after the instructions and attendees, with output constrained to a schema written enums first and free text last, as structured outputs sets out.
action_items[] owner: enum(attendee ids, "unassigned") due: date | null
task: string quote: string
decisions[] text: string quote: string
summary string, at most 150 words, last
The owner enum is built per meeting from the invite, so the model cannot invent a person, and every item carries its quote. The model has no tools: code sends the notes to the invite's recipient list, never to an address the transcript mentions.
Minutes 20–30: the eval
The round is usually lost here, by skipping it. The set: 100 past meetings labelled with action items, owners and decisions. Reading a 6,000-word transcript at about 250 words a minute takes 24 minutes and labelling it six more, so 100 meetings is 50 hours — the number that sets the launch date, far above the 1M input tokens a run costs. Slices of 20 or more: under 15 minutes, over 60, six or more speakers, customer calls, poor audio.
One grader per field. Owner match is exact and quote provenance is a substring test, so code grades both. Whether a generated task means the same as a labelled one is a judgment, so a model grades it, calibrated on 50 human-labelled pairs first. Then what the set can detect: owner precision near 0.9 on 100 meetings has a standard error of about 3 points, so a gate catches a 6-point drop, not a 2-point one, by the arithmetic in regression gates.
Minutes 30–35: the gates
Code stops the model in three places, each costing milliseconds beside a 20-second call. Input: a transcript under 300 words gets no notes, and one over budget fails loudly rather than being truncated. Output: the schema, then invariants — every owner an attendee, every due date on or after the meeting, every quote verbatim in the transcript — with one retry naming the failed check, then the draft goes to the organiser marked unchecked. Action: notes to an internal list send automatically; a list with an outside address waits for the organiser's click.
These are gates, not filters, each a check in code with a logged verdict. If 30% of meetings include an outsider, that is 3,000 clicks a day, each by the organiser of its own meeting, and no review team.
Minutes 35–40: trade-offs
Name each rejected design in one sentence. No retrieval: the p95 transcript is a tenth of the window. No agent: there is nothing to decide between calls. No fine-tune: no 10k labels, and nowhere near the volume that pays for one. Then the flip: if product asks "what did we decide about pricing across this quarter's meetings?", the corpus grows by 10,000 × 8k = 80M tokens a day, and retrieval becomes the design of a different feature.
In an interview
What is being tested is whether every box traces to a number, with fifteen minutes left for the two steps an LLM feature adds. Announce the clock once — "constraints, numbers, mechanism, eval, gates" — and keep it. Say the transcript's size before naming any component: that one number removes retrieval, chunking and the loop, and frees the minutes for the eval.
The mistake that loses points is twenty minutes on an agent architecture and none on the eval. "How would you know it works?" then arrives with three minutes left. A close second is a send tool in the model's hands.
Check yourself
1. Ten minutes in, the interviewer says meetings now average three hours. What changes on the board?
180 × 150 = 27,000 words, about 35k tokens, 37k with the prompt: under a fifth of the window, so the mechanism stands, and 1,500 tokens out is 30 s of decode, still inside five minutes. The eval changes most: reading and labelling a three-hour transcript takes about two hours, so 100 meetings is 200 hours. Label 30-minute segments, or shrink the set and say what it can no longer detect.
2. The interviewer asks why you would not run this on a tier one-fifth the price. Decide.
The transcript is the input and cannot be trimmed, so a cheaper tier is the largest lever left. Run the 100-meeting eval on it first, about 1M input tokens. Route the slices where owner precision stays within the gate's 6-point resolution, and keep the larger tier for any slice that drops.
3. Owner precision is 0.93 overall but 0.84 on 25 customer calls. Do you launch for customer calls?
On 25 items the standard error is √(0.84 × 0.16 ÷ 25) ≈ 7 points, so the 9-point gap is suggestive, not proven. Launch internal meetings. Customer calls already wait for the organiser's click, since an outside address is on the list; launch them behind that gate and label 50 more, about 25 hours, before removing it.