The model as a component · Tokens and sampling

Summarising a 40-page contract

Size one contract in tokens and decide between one call and a chunk-and-merge pipeline, with the month costed.

estimatemediumExercise · 3 min · 9 of 77

On this page

A legal-tech team wants a summary of every contract before the reviewer opens it. The decision turns on one count the candidate has to do before anything else.

The situation

A legal-tech team receives 300 commercial contracts a day, 40 pages each at roughly 500 words a page. Product wants a 400-word summary per contract before the reviewer opens the file, which today happens within 30 minutes of upload. The summary has to carry the cross-references a reviewer relies on — which clause caps which, which schedule a defined term points at — because a summary that misses them is worse than none. The team has a 2k-token system prompt that fixes the summary's shape and tone. Nothing is cached yet, nothing is chunked yet, and the budget owner wants a monthly figure before approving the feature.

Course constants, restated for the arithmetic: assume ¾ of a word per token (1.3 tokens per English word), a 200k-token window, P_in = $3 and P_out = $15 per million tokens, and a decode speed of 50 tokens/s. The month is 30 days, and the team works from the lesson on tokens and the context window.

The question

One call per contract or a chunk-and-merge pipeline — and what does a month cost?

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Count first. 40 pages × 500 words = 20,000 words × 1.3 ≈ 26,000 tokens; call it 27,000 with headings and tables, which tokenise worse than prose. Add the 2k system prompt: ~29k in. That fits one window with most of it to spare, so one call per contract, and one call keeps the cross-references between clauses that chunking loses — a chunk holding clause 14 cannot see that it caps the liability clause 3 created.

Cost per contract: 29k × $3 per million ≈ $0.087 in. A 400-word summary is ~550 tokens out × $15 per million ≈ $0.008. Together ≈ $0.10 a contract; 300 a day → $30 a day → ~$900 a month. Input is nine-tenths of it, so the lever, if one is ever needed, is on the input side.

Latency: prefill of 29k tokens is a few seconds; decoding 550 tokens at 50 tokens/s ≈ 11 s. Fifteen seconds against a 30-minute expectation, so run it in the background at upload and write the summary to the record. No streaming — nobody is watching.

Caching: the shared prefix is the 2k system prompt, 2k of 29k ≈ 7% of input, and each contract is sent once, so caching is not a cost lever here.

The answer flips to chunking only when a contract exceeds the window — at ~650 tokens a page, around 300 pages — and then as a map-reduce over a clause index, each chunk carrying the index so the merge can resolve a cross-reference, never as blind 4-page splits.

Rubric

converts pages to tokens with the arithmetic shown, and adds the system prompt before deciding it fits
chooses one call and says what chunking would lose — the cross-references between clauses
costs a day and a month, input and output separately
states the latency and whether the user path needs streaming
says what caching can and cannot save here, with the 2k-of-29k ratio
names the condition under which the answer flips to chunking, and what shape the chunking takes

Where answers go wrong

"Split into ten 4-page chunks, summarise each, then summarise the summaries." Ten calls where one fits, and the merged summary cannot see that clause 14 caps the liability clause 3 created — the candidate never checked whether the document fits.

"Stream the summary so the reviewer sees it arrive." The reviewer opens the file 30 minutes after upload; a 15-second background job is already done. Streaming adds a client path, a partial-output state and a reconnect story to a feature that nobody is watching.

"Cache the contract so the second pass is cheap." Caching pays when a prefix is resent; each contract is sent once, and the shared 2k is 7% of the input. The candidate reached for a mechanism without checking what repeats.