Design for a model that might be wrong · Being wrong

The confidence badge

Test an "87% confident" badge against the golden set and a pilot, decide whether it ships, and say what each draft shows in its place.

critiqueeasyExercise · 3 min · 40 of 59

On this page

A mock puts "87% confident" on every draft, and a two-week pilot has already measured what it does to the people reading it. This is the guideline from designing for "the model might be wrong" about showing what the model did, applied to a number that shows something else.

The situation

The drafting feature at one tenant: 1,200 agents, 48,000 drafts a day, 92% acceptable on the 200-case golden set and 0.5% wrong on a fact, one draft in 200. A wrong reply that reaches a customer costs a 20-minute escalation, and the contract has a clause on factual errors. The design lead's mock shows a badge on every draft, such as "87% confident", taken from a score the model returns with each draft. Three pieces of evidence are in. On the golden set, the 150 drafts scored 85% or higher were acceptable 138 times; the 50 scored lower, 46 times. In the pilot, 40 agents with the badge sent 51% of drafts unchanged, against 40% before, and edits that changed a number, date or price fell from 0.5% of drafts to 0.2%, while the golden set's factual rate stayed at 0.5%. Separately, the one draft in ten whose best article only just cleared the retrieval match threshold is acceptable 70% of the time; the rest, about 94%.

The question

Decide whether the badge ships, and say what each draft shows in its place.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

The badge does not ship. It fails as information and as design.

As information: 138 of 150 is 92% and 46 of 50 is 92%. Drafts scored high and drafts scored low are acceptable at the same rate, so the score separates nothing, and a draft labelled 87% that is right 92% of the time is not even calibrated.

As design, the pilot shows what it does to the reader. Fact edits fell from 0.5% to 0.2% while the factual error rate held at 0.5%, so agents stopped catching 0.3 points of wrong facts, three in five. Rolled out to the tenant's 48,000 drafts a day, that is about 144 wrong facts reaching customers daily, each a 20-minute escalation, 48 agent-hours a day, under a clause about exactly this. The 11-point rise in unchanged sends is automation bias, not trust: the acceptance metric improved because checking stopped.

In its place, three things. Sources: each draft lists the articles it used, and each price, date and policy line links to the sentence it came from, so a check takes a glance. A flag shown only where a measured signal separates: "Weak source, check the facts" on the one draft in ten whose best article only just cleared the threshold, acceptable 70% of the time against 94%, a 24-point gap the agent can act on. And a one-click report on any fact span, feeding the golden set.

What would change this: a score that, re-binned on 2,000 drafts, separates acceptable from unacceptable by 10 points or more and stays calibrated. Even then it becomes a two-level flag, never a percentage.

Rubric

Tests the badge's score against the golden set and shows it separates nothing: 92% in both bins.
Converts the pilot's fall in fact edits into wrong facts sent per day, about 144, and prices them in escalation time.
Reads the rise in unchanged sends as reduced checking, using the fact-edit count beside it rather than the acceptance figure alone.
Replaces the number with things the agent can act on: per-fact sources, and a flag only where a measured signal separates, 70% against 94%.
Names the evidence that would let a score back on the screen, and the form it would take.

Where answers go wrong

"Keep the badge but show high, medium or low instead of a percentage." Three bins of a score that separates nothing still separate nothing, and the pilot's automation bias came from the badge being there, not from its precision.
"Ship it: unchanged sends rose from 40% to 51%, so the North Star input moved." The input rose because agents stopped checking; about 144 wrong facts a day reached customers to buy it. An acceptance figure read without the fact-edit count beside it rewards exactly this.
"Replace the badge with a line saying AI can make mistakes." It moves liability, not error, and a notice on every draft is read on none. The half-point factual rate is unchanged and the agent still does not know which sentence to check.