Context assembly
Read a token budget off the cost line, fill it with three clauses filtered to the term in force, and show why the whole policy set does not go in.
On this page
The model reads what the assembler hands it and nothing else. For the triage pilot the assembler takes one claim and the 340-page policy set and returns about 4,300 tokens, in a fixed order, under a budget the definition of done has already set. Which three clauses arrive, and from which policy term, moves agreement further than any sentence in the instructions.
The budget is the cost line, read backwards
The definition of done caps cost at 0.005r a claim, where r is the contracted per-million-token input rate and output is priced at 4r. Read backwards, that line is a token budget:
output 160 tokens × 4r per million ≈ 0.00064r
input 0.005r − 0.00064r ≈ 0.00436r → 4,360 input tokens
Sizing the deployment has already spent it. At the course's assumed 0.75 words per token:
| Band | Words | Tokens | Who controls it |
|---|---|---|---|
| Instructions, task restated last | 200 | ≈ 270 | you |
| Four worked examples | 500 | ≈ 670 | you, from their claims |
| Three retrieved clauses | 900 | ≈ 1,200 | the retriever |
| Claim form and adjuster note | 1,600 | ≈ 2,130 | the customer |
| Output, reserved | 120 | ≈ 160 | the schema |
Input is about 4,270 tokens against 4,360: under 100 tokens of headroom. "Add the claim history" now means shrinking another band or moving the cost line, which is a conversation with the sponsor, not a code change.
The order, and why it is this one
The prototype assembled the prompt in the order the code fetched things: the claim, the keyword search's sections in score order, then the instructions. Claimant prose and insurer wording ran together, and the sections carried no clause numbers to cite. The pilot's order:
- Instructions: what the record contains and the citing rule, about 240 tokens.
- The four examples.
- Three clauses, each with clause number and heading path, fenced as the insurer's wording.
- The claim form and adjuster note, fenced as the claimant's and adjuster's text.
- Two sentences restating the task, about 30 tokens.
The first two bands are identical on every call, so a prompt cache, where the placement offers one, can match them. The clauses come before the claim because they are the menu the model chooses from; the fences say which text is wording and which is a claimant's prose, the first defence in failure modes on real data. The restated task is the cheap half of the lost-in-the-middle advice in tokens and the context window; 1,200 tokens of clauses has no middle to get lost in.
Chunking their documents
The retriever returns only what the chunker made, and the policy set has joints: numbered sections (4 Exclusions, 4.2 Escape of water), endorsements issued as separate documents, one wording per line of business per term. The rules from chunking, applied to their numbering:
- Split on section numbers, never across one: half of 4.2 and half of 4.3 cannot be cited as either.
- Cap a chunk at 400 tokens so three fit the band; a longer section splits at paragraph breaks and keeps its number with a suffix, 4.2#1 and 4.2#2.
- Make the chunk id the clause number handlers use. The gold set says "4.2", so the clause is scored by string comparison; a position id shifts whenever the wording is reissued.
- Prepend the heading path, so a chunk that opens "within 14 days" still has a subject.
- Store line of business, term and source document on every chunk.
A paragraph over the cap stays whole, so the cap alone does not hold the band: the builder adds clauses in rank order, heading paths included, until the next would pass 1,200 tokens, and counts the claims whose top clause alone is over.
Assume 400 words a page: the set is 340 × 400 = 136,000 words ≈ 181,000 tokens, at least 450 chunks at the cap. The exercise chunk the policy manual writes the splitter, trap included: a body line beginning "14 days after the leak" is not section 14.
Filtering comes before ranking. The claim's term, resolved from the loss date as in schemas, identity and joins, fixes the line of business and the wording version. Assume the 340 pages split as 50 motor, 60 property, 180 commercial and 50 of endorsements: a motor claim is ranked against 50 pages, and module 4's two-wordings failure cannot happen. Endorsements are their own documents, linked to the terms they modify; whether one competes for the three slots or holds its own is for the per-segment eval.
Retrieval against putting everything in
The sponsor's week-2 question is fair: the model reads long documents, so why not send the whole set and skip the index? With the same 3,070 tokens of instructions, examples and claim on every call:
| Wording in the prompt | Input tokens | Per claim | Against the line |
|---|---|---|---|
| The whole 340-page set | ≈ 184,000 | ≈ 0.185r | 37 times |
| The claim's line, motor | ≈ 30,000 | ≈ 0.030r | 6 times |
| Three retrieved clauses | ≈ 4,270 | ≈ 0.0049r | inside |
The whole set is 1,200 × 0.185r ≈ 222r a day against about 6r. The cost line rules it out before the window is consulted: the clause band is 1,200 tokens, about 900 words, so this budget affords a whole document only if it is about two pages. Caching does not rescue it: at an assumed cached read of a tenth of r, the whole set is still about 0.022r a claim, four times the line, and the prefix stops being fixed once the term filter picks a wording per claim.
What long context does and does not remove
Long context removes the retrieval miss, chunk-boundary errors and an index to keep current, which for a short wording can be the whole argument.
It does not remove the cost, which is linear in tokens, or the reading time: prefill grows with input, and 184,000 tokens is 43 times the retrieved call's input, against a 20 s p99. It does not remove the wrong-term problem, so the filter survives anyway, or the citation check: with 450 candidate clauses, the model can cite a commercial clause on a motor claim, and a validator can confirm the id exists but not that it applies.
And it blinds the eval. With retrieval, recall at 3 on the 200 gold cases is computed in seconds without calling the model. With everything in, a miss can no longer be split into "retrieval did not find it" and "the model did not use it".
In an interview
The deep-dive asks how the policy wording gets into the prompt, testing whether you size before you choose and take the budget from the cost line rather than the window. A strong answer: "The cost line gives 4,360 input tokens; the claim takes 2,130 and the instructions and examples 940, so the wording gets 1,200 with 90 to spare: three clauses of 400, filtered to the term in force, cited by clause number." What the customer notices is that the clause numbers in the record match their own PDFs.
The mistake that loses points is "the window is big enough, so put the whole manual in", said without the multiplication. 1,200 claims × 184,000 tokens is about 222r a day against 6r, and the follow-up about a renewed policy's two wordings has no answer.
Check yourself
1. Week 4: the gold clause is in the top 3 for 97% of motor cases and 64% of commercial. The team proposes top 6 for every claim. Estimate the cost and decide.
Top 6 adds 1,200 tokens, about 0.0061r a claim: over the line everywhere to fix one segment. First find where the commercial misses sit. If the gold clause is in an endorsement the index never held, no
kfixes it: index the endorsements. If it is indexed but ranked fourth to sixth, widenkfor commercial only: 0.8 × 0.0049r + 0.2 × 0.0061r ≈ 0.0051r blended, a 3% overrun the sponsor signs or refuses.
2. A specialist insurer sends 80 claims a day against one 20-page wording, and its cost line is 0.02r a claim. Put the wording in whole, or retrieve?
20 × 400 = 8,000 words ≈ 10,700 tokens, plus 3,070 of instructions, examples and claim, ≈ 13,700 input: about 0.014r a claim with output, inside the line, about 1.2r a day. Put it in whole, after resolving the term, and keep the citation check. What flips it is the wording outgrowing the line: 0.02r affords about 16,300 tokens of wording, roughly 30 pages.
3. Next January the insurer reissues 60 pages of property wording for renewing policies. What happens to the index and to the gold labels?
60 × 400 = 24,000 words ≈ 32,000 tokens, roughly 80 new chunks under the new term. The old term's chunks stay; claims against old terms arrive for years. With clause-number ids and a term on every chunk, the gold labels still score by equality; with position ids, every label after the reissued pages shifts, and the QA lead's 200 cases need labelling again. Version by term; never overwrite.