Workflows, state and the harness · State and memory

The prompt that carries the whole CRM

Find why a CRM export pasted into every prompt produced three wrong replies, decide whether caching fixes it, and price the lookup that replaces it.

critiqueeasyExercise · 3 min · 25 of 75

On this page

A CRM pasted into the system prompt looks like memory, because the model "knows" every client. It is a week-old snapshot read whole on every call, and memory and state for long-running automations is the lesson this practises.

The situation

The lead handler drafts a reply to each of the agency's 400 inbound leads a month, about 18 a working day, arriving through the day. A contractor's version skips the CRM lookup step. Its system prompt carries 1,200 tokens of instructions plus the whole CRM export, 1,800 contacts at about 60 tokens each, 108,000 tokens, refreshed every Monday "so the model remembers every client". Each call adds the lead's form and email, about 300 tokens, and returns a 250-token draft.

Last month three drafts went wrong. A reply to Priya Shah mentioned Priya Nair's open proposal. A client who cancelled on Tuesday got a welcome-back note on Thursday. A reply quoted a discount from a note another account manager had written. The contractor proposes prompt caching as the fix. Assume the large model at $2 per million input tokens and $8 per million output, a cached prefix at one-tenth of the input price, a 200k-token window, and that a lookup by email returns one record of about 200 tokens.

The question

Find the defects in this design, decide whether caching is the fix, and say what replaces it, with both designs priced per month.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Price it first. Per call, 1,200 + 108,000 + 300 = 109,500 tokens in and 250 out. A month is 43.8M in at $2, $87.60, plus 0.1M out at $8, $0.80: about $88.40.

Caching cuts the bill, not the defects. If every call hit a warm cache, 43.68M cached tokens at $0.20 is $8.74, about $9.80 a month in all. At 18 leads spread over a day, many calls arrive after an entry that lives for minutes has expired, so even that is optimistic. None of the three failures was about price.

Each failure has a cause the design created. The wrong Priya: the model matched a first name across 1,800 rows, where an email is an exact key. The welcome back: the export is up to a week old, where a lookup reads the CRM at the moment of the call. The leaked discount: every call puts every client's notes in front of the model, and sends them all to the provider 400 times a month.

Replace the export with the step it skipped: code looks up the sender's email before the model step, passes the one record without its internal notes, and sends a no-match to a person. The provider sees one record, not 1,800. Per call that is 1,200 + 300 + 200 = 1,700 in, so 0.68M at $2 is $1.36, plus $0.80 out: $2.16 a month, about 40 times cheaper than now and 4.5 times cheaper than cached, with each failure gone by construction. Build the lookup as tool design describes.

Cross-client context earns a place only if reviewed drafts show it missing, say account managers adding it by hand; it then arrives as a typed query, never as the export.

Rubric

Prices the current design per call and per month, with input and output separately.
Prices the cached version and says why it fixes none of the three failures.
Traces each failure to its cause: a name match across 1,800 rows, a week-old snapshot, every record on every call.
Replaces the export with an exact lookup by email before the model step, and prices it.
Names what reaches the model provider in each design.
Says what evidence would justify cross-client context, and in what form it would arrive.

Where answers go wrong

"Cache the prefix and move on." It takes about $79 off the bill and leaves all three replies exactly as wrong. The reviewer scores whether the answer reads the failures before the invoice.

"Put the CRM in a vector store and retrieve the five most similar contacts." A similarity search on "Priya" returns both Priyas. The email is an exact key, and an exact key is a lookup, not a retrieval problem.

"Refresh the export daily." It swaps a week-old snapshot for a day-old one, leaves the wrong match and the leaked note untouched, and keeps the bill at $88.