RAG against fine-tune as a product trade-off
Decide between retrieval and fine-tuning on freshness, citations, per-customer isolation and update cost, with a help centre and a house tone worked.
On this page
Retrieval puts documents into the prompt at request time; fine-tuning changes the model's weights with examples before any request. Engineers argue about them as techniques. For a PM they are answers to four product questions — how fresh must the answer be, must it cite, must one customer's data stay out of another's answers, and what does an update cost — and the two worked cases below come out on opposite sides.
What each one changes
Retrieval-augmented generation, RAG in Lewis et al.'s 2020 paper, keeps the model as the vendor shipped it and changes what it reads. The help centre is split into chunks, each chunk is turned into a vector, and at request time the ticket is turned into a vector too and the five nearest chunks are pasted into the prompt. The model answers from what is in front of it and can say which chunk it used. The developer's guide to LLMs covers the mechanics; context assembly in the FDE course covers what goes wrong when the chunks are the wrong ones.
Fine-tuning takes a few thousand example pairs — a ticket and the reply you wish the model had written — and adjusts the weights so the model writes like the examples without being shown them again. Nothing is pasted into the prompt. The model cannot cite an example it absorbed, and it knows nothing that happened after the examples were collected.
The four questions
Freshness. A retrieved answer is as fresh as the last index run; a fine-tuned answer is as fresh as the last training run. Index runs cost minutes and under a dollar; training runs cost hours and an eval cycle. If the source changes weekly, retrieval; if it changes yearly, either.
Citations. Retrieval can return the chunk id with the draft, so the agent sees "per article 4471, section 2" and can check it in one click. A fine-tuned model produces text with no provenance. For anything customer-facing with a contract clause about factual errors, a citation is the difference between a draft the agent can verify in ten seconds and one they have to re-research.
Per-customer isolation. A vector store per tenant means tenant A's chunks cannot appear in tenant B's prompt, by construction. A model fine-tuned on both tenants' tickets has both in its weights and no way to keep them apart. One fine-tune per tenant is possible and is a model to deploy, monitor and re-train per customer — the FDE's problem in where the model sits, and not one most products should take on.
Cost to update. Retrieval: re-embed the help centre, 3,000 articles of about 800 words at 1.3 tokens a word, about 3 million tokens a week — $0.31 at an assumed $0.10 per million embedding tokens. Fine-tune: collect new examples, train, re-run the 200-example golden set, compare, decide; a day of someone's time at minimum, and a repeat every time the vendor ships a new base model.
Worked: 3,000 help articles updated weekly
The support tool's tenants each have a help centre of about 3,000 articles, and about 70 change a week — prices, policy wording, a new product line. The drafts have to reflect this week's policy and the agent has to be able to see which article the draft leaned on.
Retrieval. Freshness is weekly or better, so the index runs weekly at least (how stale is too stale sizes the interval). Citations come free. Isolation is one store per tenant. Cost per request: the five chunks share with the ticket the 400 tokens of the 1,500-token prompt that are not the system prompt, so at most $0.0012 at an assumed $3 per million, under $240 a day on 200,000 requests; retrieval itself adds about 80 ms, invisible beside the model's 2 s. Fine-tuning here would mean a training run per tenant per week, no citations, and a model that confidently quotes last week's price. Retrieval, and it is not close.
Worked: one tone across a million replies
The same tenant wants every reply to sound like the company — short sentences, no apologies in the first line, the sign-off they use. Tone is not a fact in any article; there is nothing to retrieve.
The prompt alternative is a 600-token style guide in every call: 600 × 200,000 = 120 million tokens a day, $360 a day at $3 per million, $10,800 a month — or about $1,100 a month once the cached prefix prices those tokens at a tenth. A fine-tune on 3,000 example replies learns the tone once, adds no tokens to a request, and is re-done only when the base model changes, perhaps twice a year. Read one line first: some vendors price a tuned model's tokens above the base model's, and on the $54,000 bill a 2% premium is the whole $1,100. With no premium the two are close enough that the deciding factor is quality on the eval: a fine-tune usually holds a tone more consistently than a prompt does across a million replies, and consistency is the requirement. Fine-tune at a premium you have read, with the style guide kept in the eval rubric so the drift is measurable.
Both is common
The drafting feature ends up with both: a model fine-tuned for tone, reading retrieved chunks for facts. Each answers the questions the other cannot. The product decision is not "which" but "which question does each one answer", written down so that when the help centre moves to daily updates the index interval changes and nothing else does, and when the vendor ships a new base model the fine-tune is re-run and the index is not touched.
The combination has one cost worth naming: two things can now drift. A model update can shift tone; an article change can shift facts. Evals as the feature's test suite scores them as separate failure classes for exactly this reason.
In an interview
What is being tested is whether you reach for the four questions before the two techniques. Say them in order — freshness, citations, isolation, cost to update — and answer each for the case in front of you with a number: "changes weekly, so a weekly index; customer-facing with a contract clause, so citations; multi-tenant, so a store per tenant; re-embedding is under a dollar, retraining is a person-day." Then say "both" if both is the answer, and say which question each one answers.
The mistake that loses points is choosing fine-tuning to teach the model facts — a fine-tuned model does not know this week's price and cannot say where last week's came from. The second is choosing retrieval for tone, which pastes the same style guide into every call and still drifts. The third is missing isolation entirely; an interviewer who hears a shared fine-tune across tenants hears a data leak waiting for a security review.
Check yourself
1. A tenant's help centre changes daily rather than weekly, and the eval's time-sensitive slice drops 4 points by day three. What changes, and what does it cost?
The index interval, from weekly to daily or better, and nothing else in the design. Cost: a daily re-embed of the whole help centre, $0.31 a run at the assumed $0.10 per million, about $9 a month — plus the eval run that confirms the slice recovered. If the drop is in the facts class only, the fine-tune is untouched.
2. Legal asks for every draft to carry a reference the agent can check. The current build is fine-tune only. What do you propose and what does it add per request?
Add retrieval for the factual content: up to 400 tokens of chunks per call, at most $0.0012 at $3 per million, $240 a day, and about 80 ms. The fine-tune stays for tone. The draft now returns article ids with the text; the screen shows them as links. Say the per-request figure, because legal's requirement has just moved the cost line in the PRD.
3. The vendor ships a new base model. The tone fine-tune was trained on the old one. Which of the two paths do you re-run, and in which order?
Re-run the golden set on the new base model with retrieval and no fine-tune first: that tells you whether facts and citations still pass, which is the blocking class. Then re-train the tone fine-tune on the new base and re-run the tone slice. The index is not touched; nothing about the articles changed. Ship when both slices are back at threshold, not when the first one is.