The tech a PM needs · Evals

How stale is too stale

Turn an eval's decay with index age into wrong drafts and dollars, and pick the refresh interval where staleness and re-indexing cost least together.

estimatehardExercise · 3 min · 35 of 59

On this page

RAG against fine-tune chose retrieval for a help centre that changes every week and left the index interval open. The eval can measure what a day of staleness costs, and the interval follows from that number.

The situation

One tenant: 1,200 agents, 48,000 drafts a day. Its help centre has 3,000 articles, about 2.4 million words, and about 70 articles change a week, edited throughout the working week rather than in one batch. The retrieval index is rebuilt every Sunday night by re-embedding the whole help centre. Assume 1.3 tokens a word and $0.10 per million embedding tokens. The eval team re-ran the 200-case time-sensitive slice, tickets whose answer is a price, a date or a policy line, against copies of the index 0, 1, 3 and 7 days old: 95%, 93%, 89% and 81%. Those tickets are 10% of the tenant's traffic, and every stale answer is a factual error; the launch gate blocks above 0.5% factual errors. Agents catch three in four wrong drafts at 20 seconds each; the fourth is sent and comes back as a 20-minute escalation. Assume $30 an hour loaded.

The question

Pick the refresh interval for this tenant's index, showing for each candidate what staleness costs a month and what refreshing costs.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Read the slope off the eval: 95% fresh to 81% at seven days is 2 points per day of index age, and the 1- and 3-day points sit on that line. Edits land through the week, so under an interval of T days the index is on average T/2 old and the slice loses 2 × T/2 = T points.

A wrong draft costs ¾ × 20 s + ¼ × 20 min = 5.25 minutes, about $2.60, nearly all of it the sent quarter. The slice is 4,800 drafts a day. A full re-embed is 2.4M words × 1.3 = 3.1M tokens, about $0.31 a run.

Interval	Points lost	Wrong a day	Staleness a month	Refresh a month	Total
Weekly	7	336	$26,200	$1	$26,200
Daily	1	48	$3,740	$9	$3,750
Hourly	0.04	2	$160	$225	$380
10 minutes	0.007	0.3	$26	$1,350	$1,370

Weekly fails the gate on staleness alone: 336 of 48,000 drafts is 0.7%. Daily passes at 0.1% and still costs $3,740 a month. Hourly is the minimum at about $380, and it is a schedule change, not a project, because the job already exists. Below an hour the re-embedding costs more than the staleness it removes.

Decision: hourly, with the slice re-run monthly against index copies of known age, to check the slope still holds.

What would move it: an incremental index that re-embeds only the ten or so articles changed each day, which makes a run cost a fraction of a cent and moves the answer to every few minutes; or a tenant whose prices change daily, which steepens the slope.

Rubric

Reads a slope off the eval, 2 points a day, and uses the average index age, T/2, not T.
Prices a wrong draft with both the caught and the sent share, about $2.60, and notices the escalation is nearly all of it.
Sizes a re-embedding run from words, tokens and the assumed rate: about $0.31.
Tabulates at least three intervals with staleness and refresh cost side by side, and finds the minimum near hourly.
Checks the current weekly interval against the 0.5% factual-error gate.
Names what would move the answer: incremental indexing, or a steeper slope.

Where answers go wrong

"Weekly, because the help centre changes weekly." The changes arrive across the week, so a Sunday rebuild leaves the index 3.5 days old on average: 336 wrong facts a day, 0.7%, which fails the gate before any cost is counted.
"A wrong draft costs the agent 20 seconds." That prices only the three in four that are caught. At $0.17 a wrong draft, daily and hourly both come to about $240 a month and the answer stops at daily, but the sent quarter is $2.50 of the $2.60, and it is what makes hourly worth $3,370 a month more.
"Re-index on every edit; fresher is always better." With a full re-embed per run, every ten minutes costs $1,125 a month more than hourly to remove the $130 of staleness hourly leaves. Fresher has a price until the index is incremental, and the answer should say which change makes it free.