Retrieval, and when not to · Choosing the mechanism

The FAQ that should have been a prompt

Find what a retrieval pipeline over a 60-entry FAQ gets wrong, cost the alternative of a cached prompt, and name the condition that would bring retrieval back.

critiquemediumExercise · 3 min · 38 of 77

On this page

A support bot over a short FAQ misses one question in five, and the team is tuning the embedding model. The corpus was small enough to read whole.

The situation

A software company's support bot answers questions from a 60-entry FAQ, about 120 words an entry — 7,200 words, roughly 9,400 tokens at 1.3 tokens per word. The pipeline the team built: each entry is a chunk, embedded and stored in a vector database; each question is embedded (about 40 ms and one extra call), the top 3 entries are put in a 600-token system prompt, and the model answers. On a 200-question labelled set the bot says "I do not have information on that" for 18% of questions, nearly all of them paraphrases the embedding ranked below the top 3. A nightly job re-indexes the FAQ. The bot handles 2,000 questions a day, a 50-token question and a 120-token answer each.

Constants: a 200k-token window, P_in = $3 per million input tokens, P_out = $15 per million output tokens, cache multipliers C_w = 1.25 on a write and C_r = 0.1 on a read, applied to P_in; 30 days a month.

The question

Where does this design go wrong, what would you build instead, and at what point would retrieval be the right call again?

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

The corpus is 9,400 tokens and fits the window, so retrieval solves a problem the team does not have and creates one it does: an 18% miss rate that is entirely recall at 3 — the answer exists, the embedding ranked it fourth. No embedding model fixes a ceiling that the window can remove.

Put the whole FAQ in the system prompt, stable content first, and let the cache carry it. Per question, uncached: 9,400 + 600 + 50 ≈ 10,050 input tokens × $3 per million ≈ $0.030. With the prefix cached, the 10,000-token prefix reads at C_r = 0.1: ≈ $0.003, plus the 50-token question, plus 120 output tokens × $15 per million ≈ $0.002 — about $0.005 a question. At 2,000 a day — one every 43 seconds — the cache stays warm, so writes at C_w = 1.25 happen a few times a day, not per call. Month: 2,000 × 30 × $0.005 ≈ $300.

The retrieval version costs 600 + 50 + 3 × 160 ≈ 1,130 input tokens ≈ $0.0034, plus $0.002 out, about $0.0054 a question — the same money, with a vector database, an embedding call, 40 ms of latency and an 18% miss rate attached. The cached prompt sees all 60 entries on every question, so recall is 100% by construction; the 200-question set should show only generation errors remain.

Retrieval comes back when the corpus outgrows the arithmetic: past roughly 50k tokens the cached prefix is a visible slice of the window and the bill; when entries must be filtered per tenant, one prompt cannot hold everyone's FAQ; and when questions need facts from 2,000 help-centre articles, not 60 entries. Keep the 200-question set as the eval either way.

Rubric

sizes the FAQ in tokens and compares it with the window before anything else
attributes the 18% to recall at 3, not to the model or the embedding
costs a question both ways, with cache reads and writes at the stated multipliers
states that the cached prompt makes recall 100% by construction
names the corpus size, the filtering need or the question type that brings retrieval back
keeps the labelled set as the acceptance test for the new design

Where answers go wrong

"Raise k from 3 to 10 and switch to a better embedding model." Treats the miss rate as a ranking problem; at k = 10 it falls but does not reach zero, the embedding call and the index stay, and the candidate never noticed the corpus was 9,400 tokens.

"Fine-tune the model on the FAQ so it knows the answers." A month of work to teach 60 facts the window can hold today, with no citation and a retraining run on every FAQ edit. The symptom was missing facts; the fix for missing facts that fit is to show them.

"Keep retrieval — the prompt approach costs $0.030 a question, six times more." The uncached figure. With the prefix cached at C_r = 0.1 it is about $0.005, level with retrieval; the candidate priced the cache miss as if it were every call.