Launch
Ramp an AI feature from 5% to 100% of its traffic behind guardrails that act, size each stage in drafts, and announce what the feature does not do.
On this page
A launch is the last experiment before the feature becomes everyone's problem. The drafting feature goes from 5% of drafts to roughly 25% to 100%, held at each stage until the three guardrails from module 3 have been clean for a week, and any stage can be undone in minutes with a per-tenant flag. The announcement says what the feature does not do, because the agents who read it are the last review step between a wrong draft and a customer.
The mistake: a date and a headline
The naive plan switches every tenant on for the conference date and announces "AI now writes your replies." On day one that is 200,000 drafts. The golden set puts one draft in 200 wrong on a fact — a price, a policy, a date — the most the launch gate allows, so 1,000 drafts a day carry a wrong fact, read by agents just told the tool writes their replies. If one in ten slips past them, 100 wrong facts a day reach customers, at tenants whose contracts have a clause about exactly that. The only way back is a hotfix for everyone at once.
The fix is three numbers: how big each stage is, how long it holds, and what pulls it back.
The ramp, sized in drafts
Stages are measured in drafts, not seats, because cost and complaints scale with drafts. They are assigned by whole tenant, not by random seat: agents in one tenant share a queue, so half a team drafting changes which tickets the other half gets, the interaction problem in when an A/B test is the wrong instrument.
The gate before any stage is the eval: 92% on the golden set with no blocking failure class, as defined in evals as the feature's test suite. Then:
- Stage 1, 5%. About ten small tenants from the best-fit list in positioning, roughly 600 seats, 10,000 drafts a day.
- Stage 2, about 25%. Add the large tenant: 1,200 agents × 40 tickets = 48,000 drafts a day, 24% of all traffic on its own. With stage 1 that is 58,000 a day, 29%. The round number bends to the tenant boundary.
- Stage 3, 100%. 200,000 drafts a day.
The large tenant goes second, alone, for three reasons. It has the factual-error clause, so it should see the feature after 10,000 drafts a day have been read, not before. Its own p95 per hour is a real number: 48,000 drafts over a ten-hour day is 4,800 an hour, against about 100 for a typical stage-1 tenant. And if it breaks, its flag goes off and the other 71% of traffic never saw the feature.
How long a stage holds
The three guardrails come from AARRR and guardrails, each with its action: p95 to last token under 6 s per tenant per hour, or the stage pauses; complaints under 2 per 1,000 replies, or it rolls back one stage; cost per untouched resolution under $0.03, or drafts per ticket are capped at one and the PM is paged.
Whether a week is enough is arithmetic. Assume complaints ran at 1 per 1,000 replies before the feature. At stage 1's 10,000 drafts a day, the baseline is 10 complaints a day and the line is 20. A count of 10 varies by about √10 ≈ 3, so a doubling stands out within two or three days.
The week comes from the calendar instead. Monday carries the weekend's backlog, and the p95 that matters is Monday's at 10 am; a stage that has not seen a Monday has not seen the peak. So the hold rule is seven consecutive clean days, which always includes one.
The rollback is a per-tenant flag, tested on purpose in stage 1: turn it off for one tenant on a Tuesday afternoon, time how long drafts take to stop, turn it back on. A rollback that has never run is a hope.
Reading a stage: guardrails first, then the funnel
Clean guardrails say the feature is safe to widen, not that it works. The funnel does, read on stage 1's 600 seats over three weeks:
The step that loses the most is the fourth: of 405 agents who inserted a draft, 243 were drafting half their tickets a week later. The other 162 went back to their macros, the habit force from jobs to be done arriving on schedule. The fix is the positioning attribute, drafting from the macro the agent already picked, and it ships before stage 2: guardrails decide whether the ramp moves, the funnel decides what changes before it does. The tenants not yet on the feature are the comparison: with ten tenants this is not a powered A/B test but a pre/post on reply time against a control group, which says whether the minute is real.
The announcement names what the feature does not do
"AI now writes your replies" sets the agent's expectation at the wrong place: if the tool writes the reply, the agent's job is to send it. The announcement that goes out at each stage says four things instead:
- It drafts; it never sends. An agent reads every draft before it goes.
- It answers from your help centre and your macros. It cannot see orders, refunds or account changes, and will not offer them.
- On our test set, about 1 draft in 200 gets a fact wrong — a price, a date, a policy. Check those before sending.
- One click reports a bad draft; severe reports reach a person within an hour. The report button designs what happens after the click.
The third line is the one marketing will want to cut, and it is the line that turns an error rate into a behaviour: "check prices and dates" is something an agent can do 40 times a day. The FDE stages one customer's deployment the same way in crawl, walk, run.
In an interview
"How would you launch this?" is an execution question, and what is being tested is whether the plan has numbers in it: the gate, stages in the unit that carries cost and risk, a hold rule with its arithmetic, guardrails with actions, a tested rollback, and what the announcement says the feature does not do. "Stage 1 is ten tenants and 10,000 drafts a day, held seven clean days so it sees a Monday; complaints over 2 per 1,000 roll it back with a per-tenant flag" is a launch plan in one sentence.
The mistake that loses points is a calendar ramp — "a week at 5%, a week at 25%, then everyone" — with no sizes, no reason for the week and no action when a line is crossed. The second is a launch announcement that lists what the feature does and nothing it does not.
Check yourself
1. Day 4 of stage 1: p95 is 5.5 s, complaints 1.6 per 1,000, cost $0.024 a resolution. Sales wants 100% before a conference next week. What do you do?
Hold stage 1 to day 7, then add the large tenant, not everyone. Four days is short of the seven-day rule that guarantees a Monday, and 100% would put 190,000 untested drafts a day onto every tenant at once, including the one with the contract clause. At the conference, say it is rolling out and show the stage-1 numbers.
2. The design-partner group turns out to be 1,000 drafts a day, not 10,000. Is a seven-day stage still enough to read the complaint guardrail?
No. At 1 complaint per 1,000 the baseline is 1 a day, 7 a week, against a line of 14; a count of 7 varies by about √7 ≈ 2.6 and a count of 14 by about 3.7, so a real doubling and a bad week overlap. Two weeks gives 14 against 28, about two standard deviations apart. Hold for 14 days, or add tenants until stage 1 reaches about 10,000 drafts a day.
3. Marketing wants to drop "about 1 draft in 200 gets a fact wrong" from the stage-2 announcement because it undersells the feature. Keep it?
Keep it. At stage 2's 58,000 drafts a day, one in 200 is about 290 drafts a day with a wrong fact, and the agents are the only check; the line tells them where to look. Its number changes when the eval's fact class does, re-read after every model update; the line goes only when agents stop being the check.