Chunking
Pick a chunk size, an overlap and a split rule for a corpus, and size the index and the per-query prompt that follow from the choice.
On this page
The chunk is the unit of retrieval: it is what gets embedded, what gets scored, and what lands in the prompt. Choose it too small and the answer is split across two chunks that are retrieved separately or not at all; choose it too large and the embedding averages four topics into one vector while every query pays for text it does not need. Chunking is the cheapest decision in the pipeline to change and the one that moves recall most.
Chunk size against recall
An embedding is one vector for one chunk. A 100-token chunk is about one paragraph: its vector is specific, and a question phrased close to that paragraph finds it. But a policy that says "refunds are issued within 14 days" in one paragraph and "except for customised orders" in the next has its answer in two chunks, and the second is ranked by a sentence that never mentions refunds. A 2,000-token chunk holds the whole section, so the answer is in one place — but so are the shipping rules, the warranty and the returns address, and a question about refunds is compared against the average of all of them. Similarity falls, and the chunk that holds the answer is outranked by a shorter chunk that is only about refunds.
Size also sets the per-query bill. With k = 5, the retrieved text alone is 5 × chunk size: 500 tokens at 100-token chunks, 2,000 at 400, 8,000 at 1,600. Every one of those tokens is multiplied by P_in on every query, and the prefill before the first output token grows with them. At 1,600-token chunks the retrieved text is four times the size of the 400-token version's and, for a manual whose sections run a few hundred words, carries no more answers.
The mistake: a team picks 1,000 tokens because "more context is better", ships, and measures nothing. The practice: build the retrieval eval first — 100 questions, each labelled with the passage that answers it — then index the corpus at 200, 400 and 800 tokens and read recall at 5 for each. For prose that is organised into sections, 300–500 tokens usually wins; for code and tables the structure decides, as below. The number is measured, not chosen.
Overlap
A fixed-size split cuts sentences. The clause that answers the question starts in chunk 12 and ends in chunk 13, and neither half embeds well. Overlap repeats the tail of each chunk at the head of the next: with 400-token chunks and a 40-token overlap, any span under 40 tokens appears whole in at least one chunk. The cost is proportional: 10% more chunks to embed, store and search, and the same sentence can be retrieved twice, so the prompt assembly step deduplicates by document and position. Overlap is a patch for cutting in the wrong place; a split that follows the document's structure needs less of it.
Structure-aware splits
Documents have joints — headings, paragraph breaks, list items, code fences, table rows — and a split at a joint produces a chunk that is about one thing. The rule set is short: split on headings first; if a section is over the target size, split on paragraphs; never cut inside a code block or a table, because half a function or half a row is unrecoverable; merge sections under 50 tokens into their neighbour, because a chunk that is only a heading embeds as noise.
Then prepend the heading path. A chunk whose text is "within 14 days of delivery, to the original payment method" means nothing on its own, and its embedding reflects that. The same chunk stored as Returns > Refunds > Timeline: within 14 days of delivery… carries the context the section inherited from the page, and a question about refund timing finds it. The heading path costs ten tokens per chunk and is the single highest-return change in most pipelines.
Metadata per chunk
Every chunk carries fields beside its text: document id and title; heading path; position (chunk 12 of 170), so the neighbouring chunks can be pulled in when one is retrieved; last-updated date; and whatever access label the source had. The access label is not optional. A retrieval pipeline that indexes the HR director's folder and the all-hands notes into one store will answer a junior engineer's question with a salary table, and the model has no way to know it should not. Filter on tenant and permission before similarity is computed, the same way a tool is given the least privilege that does its job.
Metadata also lets a query skip the similarity step entirely when it can: "the 2024 version of the travel policy" filters on title and date, and the vector search runs over 12 chunks instead of 170.
The 100-page policy manual, worked
A policy manual of 100 pages at roughly 500 words a page is 50,000 words. At 1.3 tokens per word that is 65,000 tokens. Split at 400 tokens it is 65,000 ÷ 400 ≈ 163 chunks; structure-aware splitting leaves some sections short, so the index holds about 170 chunks, and a 40-token overlap adds roughly 10% more text to embed.
The index-time cost is one pass: the 65k tokens plus the 10% overlap, about 72k tokens through the embedding model, once, plus 170 vectors and their metadata in the store. The per-query cost is where the manual is paid for: k = 5 chunks is 2,000 tokens, plus a 600-token system prompt and a 50-token question, about 2,650 input tokens × P_in per question, every question. When one page of the manual changes, re-chunk that document only; a hash of each chunk's text, the trick a hash map buys, tells you which of its chunks are new, and only those are re-embedded.
The full estimate — chunks, index cost, a day of questions, and what 1,600-token chunks would do to it — is the exercise chunking the policy manual.
In an interview
What is being tested is whether you treat chunk size as a measured trade-off between recall and tokens per query, and whether you know that documents have structure. Say the three things in order: "I would split on headings to a target of about 400 tokens with a small overlap, prepend the heading path, and pick the size by measuring recall at five on a labelled set." Then size the index and the per-query prompt for the corpus you were given.
The mistake that loses points is a fixed character count with no overlap and no heading path — it cuts a sentence in two, embeds "within 14 days" with no subject, and the candidate has no number for recall to show whether it matters.
Check yourself
1. A 40-page API reference: short prose sections, many code blocks of 50–300 tokens, and tables of parameters. What is the split rule?
Split on headings, never inside a code fence or a table, and keep each code block with the paragraph that introduces it; merge the short sections into their neighbours so no chunk is only a heading. The target size follows from the blocks, roughly 300–600 tokens, and a reference is queried by exact names, so the heading path and the function name belong at the top of every chunk.
2. Recall at 5 is 0.92 at 400-token chunks and 0.94 at 800. Which do you ship?
Two points on 100 questions is two questions and inside the noise of the set. The 800-token index doubles the retrieved text — 4,000 retrieved tokens a query instead of 2,000 — for every query, forever. Ship 400, and grow the labelled set before believing a 2-point difference; if it holds at 300 questions, revisit.
3. The manual is updated on 20 pages a week. What does the weekly re-index cost?
20 pages × 500 words × 1.3 ≈ 13,000 tokens, about 33 chunks, re-embedded in one small batch; the other ~140 chunks are untouched because their hashes match. The cost is the embedding of 13k tokens a week, which is a rounding error beside one day of queries at 2,650 tokens each.