Turn a 100-page manual into a chunk count, an index cost and a daily query bill, and show what 1,600-token chunks do to the month.
estimateeasyExercise · 3 min · 36 of 77
On this page
An HR team wants a bot over the policy manual before the new-joiner cohort arrives. The index is a one-off; the queries are forever, and the chunk size sets both.
The manual is 100 pages at roughly 500 words a page, revised a few pages at a time. The bot will answer about 1,500 questions a day from employees, each question around 50 tokens, with a 600-token system prompt and an answer of about 150 tokens. The proposed pipeline splits on headings to a target of 400 tokens with a 40-token overlap, embeds every chunk, and puts the top 5 chunks in the prompt for each question. A colleague argues for 1,600-token chunks "so the model sees whole sections".
Course constants: 1.3 tokens per English word, P_in = $3 per million input tokens, P_out = $15 per million output tokens. Assume the embedding endpoint charges $0.10 per million tokens, and that a month is 30 days. Caching covers only a prefix identical across calls.
100 pages × 500 words = 50,000 words × 1.3 ≈ 65,000 tokens. At 400 tokens a chunk that is 65,000 ÷ 400 ≈ 163 chunks; heading-aware splits leave some sections short, so call it about 170. The 40-token overlap adds roughly 10% to the text embedded, about 72k tokens in all.
Index cost: 72k tokens × $0.10 per million ≈ $0.007, once. It is a rounding error, and a revision of five pages re-embeds about 8 chunks for less than a cent.
Per question at 400-token chunks: 600 system + 50 question + 5 × 400 retrieved = 2,650 input tokens × $3 per million ≈ $0.008; 150 output tokens × $15 per million ≈ $0.002; about $0.01 a question. 1,500 a day → $15 a day → about $450 a month. Caching reaches only the 600-token system prompt, since the retrieved chunks differ per question, so it trims a fifth of the input at best, and nothing if 600 tokens is under the minimum cacheable length.
At 1,600-token chunks: 5 × 1,600 = 8,000 retrieved + 650 = 8,650 input tokens ≈ $0.026, plus the same $0.002 out, about $0.028 a question → $42 a day → about $1,260 a month. Nearly three times the bill, and a prefill three times longer before the first token, for a manual whose sections are a few hundred words and already fit in 400. The number that would change the decision is recall at 5 on a labelled set of 100 questions: if 400-token chunks score meaningfully below 800 — more than the 3-point noise of a 100-item set — move to 800 and pay $0.016 a question, not 1,600.
"The index is the expensive part, so we should chunk larger to embed less." The index is under a cent either way; the queries are $450 a month, and larger chunks nearly triple them. The candidate priced the wrong half of the pipeline.
"1,600-token chunks are safer because the model sees the whole section." A section of this manual is a few hundred words and fits in 400 tokens already; the extra 6,000 tokens a query are other sections, paid for and diluting the embedding. No recall number was asked for, so "safer" is a feeling.
"Cache the chunks, then the cost falls away." The cache matches an identical prefix, and each question retrieves different chunks in a different order; only the 600-token system prompt repeats. The saving is a fifth of input at most, and the answer treated it as the whole bill.
