The model does not know your data
Size a corpus in tokens and choose between putting it in context, retrieving from it and training it in, with the arithmetic that rules two out.
On this page
A model answers from two sources: what it absorbed in training, and what is in the window for this request. Your refund policy, last week's incident report and the 2,000 articles in the help centre are in neither. Every "the model does not know our data" problem is the same question — how do the right two thousand tokens of that corpus get in front of the model for this one request — and there are exactly three ways to answer it.
Three ways to close the gap
Put it in context. The simplest option: paste the documents into the prompt. Size it before choosing it. A help centre of 2,000 articles at 3k tokens each is 2,000 × 3,000 = 6M tokens. Against the course's 200k-token window, from the four numbers, that is about thirty windows, so the whole corpus cannot be sent, whatever the prompt says. Where a corpus does fit, two costs remain. Every token is paid on every call: a 60k-token corpus in the prompt is 60k × P_in per question, before the question is asked. And a long window is not read evenly — Liu et al. found accuracy falls when the relevant passage sits in the middle of a long context, the finding behind the placement practice of instructions first or last.
Retrieve it. Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (2020), gave the pattern its name: search the corpus for the few passages likely to hold the answer and put only those in the prompt. Five chunks of 400 tokens is 2,000 tokens — three thousand times less than the corpus — so a 50-token question over 6M tokens of articles becomes, with a system prompt of a few hundred tokens, a prompt of about 2,500 tokens, read in well under a second. The corpus can change at any time; the next query sees the new version as soon as the index does.
Train it in. Fine-tuning moves the weights towards a set of examples. It teaches form reliably — tone, format, a narrow classification — and facts unreliably: the model learns the style of the documents and will state their contents with the same confidence whether or not it retained them, with no citation to check. It is also frozen at training time, so a corrected policy page needs a retraining run. The full decision table, including the cases where fine-tuning wins, is in prompting versus retrieval versus fine-tuning.
Retrieval is a search problem before it is a model problem
A retrieval pipeline has two halves. At index time: split every document into chunks, compute a representation of each chunk — a keyword index, an embedding, or both — and store it with the chunk's metadata. At query time: represent the question the same way, find the top k chunks, place them in the prompt beside the question, and generate. The model is the last step and it only sees what the search returned.
That ordering sets a ceiling. If the chunk containing the answer is not among the five retrieved, no instruction in the prompt recovers it; the model either says it does not know or, worse, answers from a chunk that looks related. The quality of the answer is bounded by recall at k, and recall is a property of the search, not of the model.
The mistake this produces is familiar. A support bot answers 30% of questions wrongly. The team rewrites the system prompt four times over two weeks: more emphasis on accuracy, a worked example, a rule about saying "I don't know". The error rate moves from 30% to 28%. Then someone labels 100 questions with the chunk that answers each and measures: the right chunk was in the top five for 80 of them, and in those 80 the model answered correctly 72 times. The model was 90% accurate when it was given the answer, and 20 of the 28 wrong answers were questions whose chunk never reached it. Retrieval was the problem, and chunking and hybrid search — not prompt wording — took the bot to 88%.
So measure retrieval on its own, with its own eval: 100 questions, each labelled with the chunk or chunks that answer it, scored on whether a correct chunk appears in the top k. Every change to chunk size, embedding model or fusion is a change to that number, and it is checked before any prompt is touched.
The search structures are the ones you already know. A keyword index is an inverted index — a hash map from each term to the list of chunks that contain it. Selecting the top k of 100,000 scores is a bounded heap, not a full sort. Vectors, the one new structure, get the next two lessons.
When the corpus fits
Retrieval is machinery, and machinery that is not needed is a liability: an embedding call per query, an index to keep fresh, and a recall ceiling under 100%. Twenty articles at 3k tokens is 60k tokens. That fits the window with room for the conversation, so put the articles in the system prompt, stable content first, and let prompt caching turn the 60k into a cached read on every call that arrives while the entry is warm. The model sees every article on every question; recall is 100% by construction.
Three numbers make the decision:
| Number | Favours context | Favours retrieval |
|---|---|---|
| Corpus in tokens | a fraction of the window | more than the window, or close to it |
| Queries per day × corpus tokens | small enough that P_in × the product is a rounding error, even uncached | large enough that paying 2,000 tokens a query instead of 60k is the budget |
| How often the corpus changes | rarely — a cache write per change is nothing | hourly — re-index the changed documents, nothing else |
The second row is the one teams skip. A 60k-token corpus at 50 questions a day is 3M input tokens a day, and at one question every half hour a cache entry that lives for minutes has often expired between them. The same corpus at 5,000 questions a day is 300M tokens a day — and retrieval at 2,500 tokens a query is 12.5M. The corpus did not change; the volume did, and the volume chose retrieval.
In an interview
The interviewer is testing whether you size the problem before naming a mechanism. Say the corpus in tokens first — "2,000 documents at 3k each, 6M tokens, thirty windows" — and let that rule out context stuffing out loud. Then say that retrieval is a search problem with its own eval, and that the model only sees what the search returns. Name the small-corpus case unprompted: "if it were twenty documents I would cache them in the prompt and skip the index".
The mistake that loses points is proposing a vector database before stating how many tokens the corpus is, closely followed by treating a retrieval miss as a prompt problem: rewriting instructions when the chunk with the answer was never in the window.
Check yourself
1. A product catalogue of 300 pages at 2k tokens each, 5,000 questions a day. Context or retrieval?
300 × 2,000 = 600k tokens: three windows, so the whole catalogue cannot be sent and retrieval is forced. If the catalogue were 60 pages — 120k tokens — it would fit, but 5,000 questions × 120k = 600M input tokens a day, 48 times the 12.5M tokens retrieval would read, and still several times it with every call a cache read at
C_r×P_in. At 50 questions a day the answer flips back to context. Both the size and the volume are in the decision.
2. The bot is wrong 30% of the time. What do you measure before changing anything?
Recall at
kon a labelled set: 100 questions, each tagged with the chunk that answers it, and whether that chunk is in the top five. The 30% splits into retrieval misses and generation errors, and the two have different fixes. Rewriting the prompt first is working on the smaller half without knowing it.
3. Prices change hourly and the question is "what does the enterprise tier cost today?"
Hourly change rules out training it in, and an hourly re-index is possible but a tool call is exact: a
get_price(tier)tool returns the current number, and retrieval is kept for the text questions — what the tier includes, what the cancellation terms say. Retrieval answers "what does the document say"; a tool answers "what is the value now".