Retrieval, and when not to · Choosing the mechanism

"Total spend in Q3" over invoices

Decide what replaces retrieval when a finance question is an aggregate over 4,000 invoices, cost the extraction once, and say what evidence would keep the pipeline as it is.

casehardExercise · 3 min · 39 of 77

On this page

A retrieval pipeline answered a finance question with a confident, precise, wrong number. The question was never a text question, and the fix is not a better retriever.

The situation

A finance team put 4,000 supplier invoices — PDFs, about 1,500 tokens each — into a retrieval pipeline: chunked at 400 tokens, embedded, top 5 into the prompt. It answers "what did the March data-centre invoice say about late fees" well. Last week the CFO asked "what was our total spend in Q3 on cloud vendors" and got $84,000, the sum of the five invoices retrieved. The real figure, found by hand in two hours, was about $1.2M across 310 invoices. The CFO now wants either a number she can trust or the pipeline switched off. Questions arriving are roughly half text lookups and half aggregates — totals by vendor, by quarter, by category — and the invoices arrive at about 80 a week.

Constants: a 200k-token window, P_in = $3 per million input tokens, P_out = $15 per million output tokens; a one-off extraction run is work that can wait, so a batch endpoint at half price is available.

The question

What do you change, what does it cost, and what evidence would persuade you to leave the retrieval pipeline exactly as it is?

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

The failure is structural: retrieval returns the top k; a total needs all n. Raising k to 310 puts 124k tokens in the prompt and asks a language model to add 310 numbers, which it does worse than a calculator, without knowing whether it had all 310. No retriever answers an aggregate.

Change the mechanism for half the questions. Extract structured fields from every invoice once — vendor, date, amount, currency, category, invoice id — with a schema, into a table. 4,000 × 1,500 ≈ 6M input tokens × $3 per million ≈ $18, plus about 100 output tokens an invoice, 400k × $15 per million ≈ $6; on the batch endpoint, about $12 for the lot. Validate extraction on 200 hand-checked invoices before trusting the table, and run it on each week's 80 new invoices as they land. Then the CFO's question is a spend(category, quarter) tool that runs SUM(amount) over the table: exact, milliseconds, and it can list the 310 invoice ids it summed. The tool takes typed arguments and is read-only; the model never writes SQL text.

Keep retrieval for the other half — the late-fees question is a passage, and the pipeline already answers it. Route by the question's shape: a sum, count, comparison or "which" over many rows goes to the tool; "what does it say" goes to retrieval.

Leave the pipeline alone only if aggregates are rare — two a month, answered by a person with the table — or if extraction accuracy on the 200-invoice check stays under 95%, in which case the table is not yet trustworthy and the CFO gets a human-run query until it is. The one unacceptable outcome is the $84,000 answer: a wrong total delivered confidently.

Rubric

explains why top-k retrieval cannot produce an aggregate, independent of k
proposes a one-off structured extraction and costs it with the arithmetic shown
answers the aggregate with a typed, read-only tool over the table, not a prompt
keeps retrieval for the text questions and states how a question is routed
validates the extraction on a checked sample before relying on it
names the evidence that would leave the pipeline unchanged

Where answers go wrong

"Increase k to 500 so the model sees all the Q3 invoices." 500 × 400 = 200k tokens is the whole window, the model is asked to add 310 figures, and nothing confirms the 310 were all retrieved. The number would still be wrong, now expensively.

"Add a prompt line: if the question is a total, say you cannot answer." Honest but useless: half the CFO's questions are totals, and a bot that refuses half the questions is switched off next week. The reviewer wanted the mechanism that answers them.

"Let the model write SQL against the invoices table." Right data, wrong tool surface: free-text SQL from the model is a write path and an injection path waiting for a malicious invoice. A spend(category, quarter) tool with typed arguments answers the same question and refuses everything else.