Metrics and analytics · Experiments

The readout

Decide whether to ship, hold or extend an experiment whose primary metric, a guardrail and a secondary metric moved in three different directions at day 21 of 30.

casemediumExercise · 3 min · 27 of 59

On this page

Three numbers moved three ways, the engineering lead wants to ship, and the test has nine days left. The decision has to be defended with the numbers on the page.

The situation

A new draft format — shorter, with the policy clause quoted — is being tested against the current one on policy-question tickets across one tenant's 1,200 agents, randomised by agent. The test was sized for 30 days at 2,000 drafts a day, 1,000 per arm, so 30,000 per arm, enough by the rule of thumb to detect a 3% relative lift on draft acceptance; the pre-registered ship rule is "acceptance up with the interval excluding zero, and no guardrail crossed". It is day 21, 21,000 drafts per arm. The readout: draft acceptance, the primary metric, 40.0% control against 41.0% variant, p = 0.04, interval on the lift +0.1 to +4.9% relative. Median time to send, a secondary metric, 120 s control against 105 s variant, clearly significant. The 7-day reopen rate, a pre-registered guardrail with a threshold of no more than +0.3 points, 10.0% control against 10.6% variant, p = 0.04. Complaint rate flat. The engineering lead reads the first two lines and wants to ship to all tenants on Monday.

The question

Ship, hold or extend — and what result on day 30 would make you reverse that decision?

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Extend to day 30, and say now what day 30 has to show. The primary metric is at p = 0.04 on a day-21 look with 70% of the planned sample, and the experiment lesson is explicit that a p-value read before the horizon is one of many looks; the interval, +0.1 to +4.9% relative, brushes zero, so the lift could be nothing. On its own that is "wait nine days".

The guardrail decides it. Reopens are up 0.6 points against a pre-registered threshold of 0.3, and a reopen is the customer saying the resolution was not one. In the North Star product form, acceptance up 2.5% relative and no-reopen down from 0.900 to 0.894, a 0.67% fall, nets to about +1.8% on untouched resolutions, so the format is not obviously bad; but the guardrail exists because reopens carry a cost the North Star does not count, a second ticket and a customer who has been told the wrong thing. The 15-second faster send is consistent with the story that agents accept shorter drafts faster and check them less. A faster acceptance of a worse reply is not a win.

Day 30 rule, written now: ship to 25% of tenants if the reopen delta is at or under +0.3 points with the full sample and acceptance's interval still excludes zero; kill if reopens stay at or above +0.5; if reopens land between, extend two weeks with the reopen split by ticket type, since the quoted-clause format may be wrong only for one intent. What would reverse an extend: the reopen rise being concentrated in one ticket type that can be excluded from the format on Monday.

Rubric

Notices day 21 is a look before the horizon and treats the p = 0.04 as one of many looks
Reads the interval, +0.1 to +4.9%, and says the lift may be nothing
Treats the crossed, pre-registered guardrail as a stop condition rather than one line among three
Nets the primary and the guardrail through the product form before deciding whether the trade is even positive
Writes the day-30 rule in advance, with both the ship figure and the kill figure
Names the segmentation that would change an extend into a ship

Where answers go wrong

"Ship: acceptance is up and significant, and sends are faster" — reads two good lines and the p-value, misses that the one pre-registered guardrail is crossed at twice its threshold, and ships a format that makes agents accept worse replies faster.

"Kill: reopens are up and that is the customer" — right instinct, wrong timing; at day 21 the reopen figure is at p = 0.04 too, and killing on a partial sample throws away the nine days that would settle it. The decision is extend with a rule, not kill.

"Extend indefinitely until it is clearer" — no rule means day 30 produces the same argument with more decimals. An extend without a written ship and kill figure is a hold with a longer name.