Prompt and context engineering on real data · Failure modes

The prompt that passed the demo and failed the pilot

Find why a prompt that scored 48 of 50 in the prototype scores 84 of the first 100 gold cases, decide what changes before week 4, and say it to the sponsor.

critiquehardExercise · 4 min · 38 of 64

On this page

The prompt scored 96% in the prototype week, the sponsor put 96% on a slide, and the first stratified reading in week 3 says 84%. The question is which of those numbers was ever true, and what in the prompt made the gap.

The situation

The claims-triage pilot from failure modes on real data. The prompt was written in the prototype week and revised nine times against the prototype's 50 claims until it cited the handler's clause on 48 of them; the demo on yesterday's claims went three for four, the miss explained. Its instructions, in full:

You are an expert claims handler with 20 years' experience.
If the claim mentions "escape of water", the governing clause is 4.2.
If the claim mentions a storm, the governing clause is 7.1.
If you are not sure, choose fast-track.
Examples: [four claims from the 50]
Reply as JSON with the route and the clause.

By mid-week 3 two senior handlers, under the QA lead, have labelled 100 of the 200 gold cases, drawn at production shares: 55 motor, 25 property, 20 commercial. The prompt cites the handler's clause on 84 of them, 84% against the 95% retrieval hit-rate line: motor 52 of 55, property 20 of 25, commercial 12 of 20. Two facts from discovery: the property wording reissued in January moved escape of water to 4.3 for renewed policies, and commercial storm cover sits in an endorsement. Fraud review is 5% of claims. The sponsor's slide still says 96%.

The question

Find the defects in the prompt and in how it was scored, then decide what changes before the week-4 reading and what you tell the sponsor about the 96%.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

The scoring defect comes first. The prompt was revised nine times against the 50 claims it was scored on, so 48 of 50 is a training score, and three of four is four cases. The first honest number is 84 of 100.

The segment split locates the prompt's defects. The phrase rules route by words in the claim, not by the retrieved wording: on renewed property policies escape of water is now 4.3, so the rule cites a clause that no longer governs, and on commercial claims "storm" fires 7.1 when an endorsement governs. That fits property at 20 of 25 and commercial at 12 of 20, while motor, which the rules never touch, holds at 52 of 55. The fast-track default turns every doubtful claim into the route where an error is a paid claim, and can hide fraud review, 5% of claims, without moving the clause hit rate. And a JSON shape with no schema, no enum and no id check gives the parser nothing to trust.

Before week 4: delete both rules; instruct the model to cite only retrieved clauses by id, endorsements over the base clauses they modify; replace the default with needs_handler and a reason; switch to the structured record from the prompt is a spec; tune on a separate 100-claim set and keep the 200 for scoring. Report by segment.

To the sponsor: "96% was measured on the claims we tuned on. 84% on 100 claims we never touched is the first real number. Motor is 52 of 55; commercial is at 60% because of two rules we delete this week."

If the commercial misses were retrieved and ignored, the prompt is the fix; if never retrieved, the index is.

Rubric

identifies 48 of 50 as a score on the tuning set and names 84 of 100 as the first real number
ties the phrase rules to the segment split, the reissued property wording and the commercial endorsement
names the fast-track default as turning doubt into the expensive error, and replaces it with a decline
separates the set the prompt is tuned on from the set it is scored on, from now on
gives the sponsor a sentence with segment numbers rather than a promise
names the evidence, retrieved and ignored or never retrieved, that decides prompt against index

Where answers go wrong

"Add rules for the commercial misses." Each rule patches one phrase in one wording version, the next reissue breaks it, and tuning against the gold set's misses turns the gold set into a second tuning set, which is the defect the answer was meant to find.

"Rewrite the persona; '20 years' experience' is doing nothing." True, and it costs about 15 tokens. It is not why commercial sits at 60%, and time spent on tone is time not spent deleting the rules and the default.

"Tell the sponsor 84% is noise around 96%." On 100 cases one case is a point: 96% would be 96, and 84 is 12 cases short, far outside a two-point noise band. It is a real gap, and calling it noise spends the credibility the week-4 reading will need.