Prompting versus retrieval versus fine-tuning
Match a stated data problem to prompting, retrieval, fine-tuning or a tool, and show the arithmetic that rules out the other three.
On this page
"The model does not do what we want" is four different complaints, and each has one mechanism that fixes it and three that waste a quarter. It lacks the facts: retrieval. It has the facts and the wrong manner: a prompt. It must do one narrow thing a million times a day, fast and cheap: a fine-tune. It is being asked for a number that only a query over every row can produce: a tool, and no amount of retrieval. The decision is made from the symptom and the arithmetic, usually in that order — prompt in an afternoon, retrieval in a week, fine-tune in a month with data collected first.
The decision table
| The problem | The mechanism | Why the others lose |
|---|---|---|
| Facts the model never saw, or facts that change daily | Retrieval | A fine-tune is stale the day a document changes; a prompt cannot hold 6M tokens |
| Tone, format, persona, a house style | A prompt, with examples | Nothing to retrieve; a fine-tune for style is a month's work to do what five examples do in an hour |
| A narrow task, 10k labelled examples, a million calls a day, latency pressure | A fine-tune | A long few-shot prompt is paid on every call; retrieval has nothing to find |
| An aggregate, a join, a count — "total spend in Q3" | A tool that runs a query | Retrieval returns the top k, and the answer needs all n |
| The corpus fits the window and rarely changes | Context, cached | An index for 60k tokens adds a recall ceiling and machinery for no saving |
Facts belong in retrieval
A model that learned your product in training would need to be retrained every time the product changed, and it would still answer without a citation. Retrieval puts the current text in the window, and the answer can quote it. The arithmetic from the first lesson applies: 2,000 documents at 3k tokens is 6M tokens, thirty of the course's 200k-token windows from the four numbers, so context is out, and a document corrected at 09:00 is in the next query's prompt at 09:01 once the changed chunks are re-embedded.
Fine-tuning on documents is the mistake that costs the most to discover. A team fine-tunes on 4,000 pages of internal documentation, and the model's answers begin to sound like the documentation — the right headings, the right terminology, the right confidence. The facts inside the sentences are correct about as often as they were before, because gradient steps over a few thousand pages adjust style far more than they install recall of a specific clause. There is no citation to check, no chunk to show, and the whole run is repeated for every edit. The month's work bought a voice.
Tone and format belong in the prompt
A prompt is an interface: role, task, constraints, output format, examples. Everything about manner lives there, and the test is whether five examples would show it. A house style does: three examples of a reply in the company's voice, a schema for the output, and the model matches it from the first call. The cost is the prompt's length on every call, and prompt caching reduces that to a cached read when the prompt is stable. Procedures that only apply sometimes move out of the prompt into a skill, loaded when they apply.
A narrow task at volume belongs in a fine-tune
Fine-tuning earns its place when three numbers line up. The task is narrow and fixed — classify a ticket into 12 categories, extract six fields from an invoice. There are around 10,000 labelled examples, and labelling is the work: at a minute an item that is about 170 hours of someone's time, before the fine-tune starts. And the volume makes the prompt expensive: a 3,000-token few-shot prompt at a million calls a day is 3G input tokens a day × P_in; a fine-tuned model needs a 300-token prompt for the same task, 300M tokens a day, a tenth of the input bill at the same P_in, and a tenth of the prefill latency before the first output token. If the fine-tuned model is also a smaller tier, P_in itself falls — the routing argument, with the eval run on the small model before the switch.
The fine-tune has its own costs. The model is frozen: a thirteenth ticket category means new labels and a new run. It still needs retrieval for facts, so it is an addition to the pipeline, not a replacement. And it is only known to work through a golden set held out from training; without one the improvement is a feeling.
When retrieval is the wrong tool
Retrieval returns the top k chunks. "Total spend in Q3 on cloud vendors" needs every one of the 310 cloud-vendor invoices in Q3, summed. Top-five retrieval returns five invoices, and the model confidently sums them: a precise wrong number. Raising k to 310 puts 310 × 400 = 124k tokens in the prompt, most of the window, and asks a language model to add 310 figures, a task it does worse than a calculator. The question is not a text question. It is a query: SELECT SUM(amount) FROM invoices WHERE quarter = 'Q3' AND category = 'cloud', exact, in milliseconds.
The mechanism is a tool over structured data: extract the fields once per invoice with a schema — vendor, date, amount, category — into a table, and give the model a spend(category, quarter) tool that runs the query. The tool takes typed arguments and returns a number, and what it takes, returns and refuses is a design in its own right. Anything a SQL tool answers exactly — sums, counts, joins, "which customers have both", "the latest" — is a tool question. Retrieval stays for "what does the contract say about termination", where the answer is a passage.
The same split applies to live values: today's price of the enterprise tier is a lookup against the system that holds it, not a search over documents that mention it.
In an interview
What is being tested is whether you match the mechanism to the symptom with a number attached, and whether you can say why each of the others loses. Walk the table: "facts that change — retrieval, because a fine-tune is stale on day two; style — prompt, because five examples do it; a narrow task at a million calls a day with 10k labels — a fine-tune, because the few-shot prompt is 3G tokens a day; a sum over all rows — a tool, because top-k cannot see all n." Then say the order: prompt first, retrieval second, fine-tune last, and only with a golden set.
The mistake that loses points is reaching for a fine-tune to teach the model facts — it buys the documents' voice, not their contents, has no citation, and is redone on every edit — and, a close second, answering an aggregate with retrieval and reporting the sum of five.
Check yourself
1. A legal team wants the assistant to write in the firm's style and cite the current version of each of 800 precedent documents. Which mechanisms, and in what order?
Two: a prompt with three examples of the style, and retrieval over the 800 documents (at 3k tokens each, 2.4M tokens, twelve windows, so context is out). Prompt first, an afternoon; retrieval second, a week with the labelled set. No fine-tune: the style is five examples' worth, and the precedents change.
2. A ticket classifier runs on 400 calls a day with a 2,500-token few-shot prompt. Fine-tune it?
400 × 2,500 = 1M input tokens a day ×
P_in, and with the prompt cached a fraction of that. A fine-tune needs 10k labels — 170 hours at a minute each — to save a bill that small. Keep the prompt; cache it; revisit when the volume passes tens of thousands of calls a day or the latency of the long prefill is the complaint.
3. "Which customers filed more than three tickets about billing last month?"
A tool: a count grouped by customer with two filters, over all of last month's tickets. Retrieval would return the most billing-like chunks, not every billing ticket, and could not count. If ticket categories are not already structured, the extraction step — a schema over each ticket, once — comes first, and that is the retrieval-adjacent work worth doing.