Prompt and context engineering on real data · Context assembly

The prompt is a spec

Write the prompt from the customer's rubric, pick few-shot examples from their claims, define a record a parser can trust, and give a refusal somewhere to go.

Lesson · 8 min · 34 of 64

On this page

A prompt is the specification the model is held to on every claim, and on a deployment the customer has already written the acceptance test: the rubric their reviewers score the 200 gold cases against. A prompt written from the demo describes what the engineer hoped the model would do. A prompt written from the rubric has, for each line that will be scored, an instruction that asks for it, an output field that records it and a check that catches it.

The first draft, and what it leaves out

The prototype's first draft, before the nine revisions that the prompt that passed the demo and failed the pilot starts from:

You are an expert claims handler. Read the claim and the policy
sections below. Suggest the route and the relevant clauses, as JSON.

The gold set carries two labels per case: the escalate-or-approve decision that building the eval set labels, and the governing clause and route, whose label instructions, written by the claims QA lead, are the rubric. They have five lines:

The governing clause is the one that decides the claim, from the term in force on the loss date.
Where an exclusion applies, the exclusion governs, even if a cover clause also matches.
Where an endorsement modifies a base clause, the endorsement governs.
The route is fast-track, adjuster or fraud review, as the triage manual defines them.
Where the wording does not decide the claim, it goes to a handler with the reason.

The first draft addresses line 4 loosely and the other four not at all. "Relevant clauses" is not "the governing clause": three relevant clauses in any order do what was asked and still do not say which one decides. Lines 2 and 3 are the cases handlers find hard, and line 5 has no path at all, so every claim gets a route whether the wording supports one or not.

One rubric line, one instruction, one field, one check

The pilot's prompt follows the rubric line by line, in the five-part shape from prompt design as interface design:

Rubric line	Instruction	Output field	Check
Governing clause	cite the deciding clause by number, from the clauses given	`clauses`	id among those retrieved; equals gold
Exclusions govern	if an exclusion applies, cite it first	`exclusion_applies`	the eval's exclusion cases
Endorsements govern	cite the endorsement over the base clause it modifies	`clauses`	the commercial cases
Route by the manual	the manual's three definitions, quoted	`route`	an enum in the schema
Undecided goes to a handler	if the clauses do not decide it, say so and why	`decline`	decline rate, by reason

The instructions come to about 200 words, the 270 tokens context assembly budgeted, and every sentence traces to a line someone will score. The route definitions are quoted from the triage manual because the model's idea of "fraud review" is generic and the handlers are scored against the manual's. And the 200 gold cases are tagged by the rubric line they exercise, so a drop arrives with an address: "exclusion cases fell from 18 of 20 to 13" is fixable, "agreement fell 3 points" is not.

Few-shot from their own examples

The four worked examples cost 670 tokens, about 15% of input, and are chosen to show one plain case and then the rubric lines the model gets wrong: one motor fast-track, one property claim where an exclusion governs, one commercial claim where an endorsement governs, and one decline where the clauses conflict.

They come from the customer's own claims, redacted with the placeholders from schemas, identity and joins, keeping the adjusters' shorthand ("EoW" for escape of water) that the model will meet in every note. They never come from the gold set: an example that is also a test case grades itself. Draw them from the week before the gold set's window and record their claim ids so the eval excludes them.

Whether the examples stay is the eval's call, as in the token budget exercise: keep them unless the 200-case set shows they lift agreement by less than the two-reviewer noise band of about 2 points.

Structured output the downstream system parses

The record is read by code: it lands in pilot.triage_output and the claims screen pre-fills the triage form from it. A regex over a free-text route field, "fast track, see 4.2", works until the model writes "Fast-track (4.2)". So the record is JSON against a schema, and field order is part of the spec:

{
  "clauses": [{ "id": "4.2", "quote": "sudden escape of water from a fixed installation" }],
  "exclusion_applies": false,
  "route": "fast_track",
  "reason": "Sudden escape from a fixed pipe; no exclusion in the retrieved wording.",
  "decline": null
}

Evidence comes before the verdict because the model decodes left to right: a route written after the clauses is conditioned on them, and one written first is a guess the clauses are fitted to. This reverses the enums-first order in structured outputs and JSON Schema, which suits a label decided from the input alone; here the route depends on which clause governs, so the clause is decoded first. The record is about 120 words, the 160 tokens reserved.

The schema enforces types and the route enum. It cannot say that each clause id is one of the three retrieved, that each quote appears verbatim in its clause, or that decline is set exactly when the route is needs_handler; a validator in code does, and the quote check is the cheapest guard against a clause that does not exist. The model never writes the claim id, the term id or the prompt version: the service stamps them, because a copied claim id attaches a record to the wrong claim.

Nor does it write the escalate-or-approve decision the gold set scores; that is derived from the record: a claim escalates when its route is fraud_review or needs_handler, or when it is over the high-value limit the service checks in code, and every other claim, fast_track or a standard adjuster, is approved, which puts about 20% of claims, 40 of the 200 gold cases, on the escalate side.

When the model refuses, and what the queue does

Three outcomes are not a draft. A decline is a valid record with route needs_handler and a reason. A refusal from the API, recorded in the stop reason that reading the response says to check first, is not retried, because the same input refuses again. An output that fails the checks is retried once with the error, then given up.

Nothing is dropped. Every path ends at a handler inside the same 15 minutes, with a draft or the reason there is none.

Each claim still reaches a handler inside the usual 15 minutes, with the reason, the fallback the four shapes built into the queue. And all three count against coverage. The definition of done allows 10% of 1,200 claims, 120 a day, to go undrafted, shared with scanned documents and missing policy numbers. A decline rate of 5% is 60 a day, half the allowance, before a scanned PDF is counted, so the decline rate is reported daily by reason, like any other budget.

The line that must never appear is "if you are not sure, choose fast-track". It turns uncertainty into the route where an error is a paid claim, and with fraud review at 5% of volume, the model can look 95% right while fast-tracking every case it should flag.

In an interview

The deep-dive hands you a prompt or asks for one, testing whether it is derived from the acceptance criteria, whether the output is built for a parser and whether "I cannot tell" has somewhere to go. Say: "Every rubric line has an instruction, a field and a check; the record is decoded evidence first; a decline is a valid output that routes to a handler and counts against coverage." The customer notices that their QA lead's wording is in the prompt, and that the prompt can be reviewed as a document.

The mistake that loses points is a prompt with no path for an undecided claim. The model is forced to choose a route; its guesses land in the agreement figure as errors, or, with a fast-track default, as fraud paid out.

Check yourself

1. In week 5 the QA lead adds a sixth rubric line: a claim reported more than 30 days after the loss goes to an adjuster, whatever the wording says. Prompt or code?

Code. The rule reads two dates already in the canonical row, loss_date and reported_at, and is right every time without a model. Apply it after the model, overriding the route with the reason recorded, so the record still shows what the wording said; tag the rule's cases in the eval. A prompt instruction would spend tokens the budget lacks and be followed most of the time; code is followed every time.

2. Week 3: declines run at 9% of claims, scanned documents at 4% and missing policy numbers at 2%. Does the pilot meet coverage, and what do you change?

9 + 4 + 2 = 15% undrafted, so coverage is 85% against 90%: 180 claims a day against an allowance of 120. Read the decline reasons before touching the prompt. If most say "wording not retrieved", it is a retrieval gap, often an unindexed endorsement, and fixing it lowers declines honestly. Declines for conflicting clauses are the prompt working; telling the model to decline less turns them into confident errors. Bringing declines to 4% reaches 90%.

3. Agreement is 93% on the 200 gold cases. The team wants two more worked examples, 335 tokens, to fix the 14 misses. Decide.

4,270 + 335 = 4,605 input tokens, over the 4,360 the cost line allows: about 0.0052r a claim. It is a swap, not an addition. Read the rubric tags on the 14 misses: if 9 are endorsement cases, replace the motor fast-track example, which teaches the model what it already gets right, with a second endorsement example, and re-run the 200. A lift under the 2-point noise band is not a lift.