Execution
Work "the metric dropped 10%" and "the pass rate fell after a model update" down a segmentation tree, and stop at the cut after which nothing changes.
On this page
An execution question hands you a number that moved and a clock. "The North Star dropped 10% this week" is the classic; "the golden-set pass rate fell after the vendor's model update" is the AI version, and it has a deadline the classic lacks, because every day on a bad build is a day of wrong drafts. The interviewer scores two things: whether you segment before you explain, and whether you stop segmenting when the next cut can no longer change what you do.
The classic: the metric dropped 10%
The weak answer explains first: "agents must have lost trust in the drafts." It is a story with no cut behind it, the mistake cohorts and funnels names. The strong answer says the tree before running any of it, and each level is one query:
- Is it real? The event behind the metric, its definition, the calendar: a holiday in one region, a client release that renamed an event, a dashboard that changed its window.
- Which input? The North Star is a product of four inputs, so a 10% fall is some mix of falls in four terms, and one query returns all four.
- Which segment? Cut the input that moved by whatever sits nearest to it: tenant, client, ticket type.
- Since when? The first bad day, against the release log.
Worked. Untouched resolutions fell from 540,000 a week to 486,000, exactly 10%. Level 1 finds nothing fleet-wide. Level 2: tickets 2,500,000, coverage 0.60 and no-reopen 0.90 are flat, and acceptance went from 0.40 to 0.36; 2,500,000 × 0.60 × 0.36 × 0.90 = 486,000, so the whole fall is acceptance. Level 3: one tenant sends a quarter of the volume. Its acceptance reads 0.24 and everyone else's still reads 0.40, and 0.75 × 0.40 + 0.25 × 0.24 = 0.36. Level 4: the fall began the day that tenant moved its agents to the desktop client.
Which sends the tree back to level 1, for that tenant only: the desktop client logs a sent draft as reply_sent, not draft_sent. The drafts were sent; the metric did not see them. The SQL lesson's first GROUP BY by tenant and client finds it in one query. The decision is an event mapping and a backfill, not a roadmap change. Each cut re-asks "is it real" for its own segment, because a fault too small to see across the fleet is a quarter of the signal one level down.
The AI version: the pass rate fell after a model update
The weekly run scored 178 of 200, 89%, against 92% last week, in the week the vendor updated the model behind the alias the feature follows. The prompt did not change. The headline invites one sentence — "the update made it worse, roll it back" — and that sentence is a hypothesis, not a finding.
Is it real? At 200 cases and 92%, the margin is about ±3.8 points (evals as the feature's test suite has the arithmetic), so a 3-point fall is inside it. The headline is the wrong instrument. The same 200 cases ran on both weeks, so compare them case by case: 10 went from pass to fail and 4 from fail to pass, net 6. Re-run both ($2 and ten minutes each) and the 10 fail again. Ten named failures you can read are stronger evidence than a 3-point headline.
What changed? List every input to the run, not the announced one: model version, prompt version, retrieval index, the set itself. If the set grew, as in working with an FDE and with research, where 92% became 88.6% entirely through ten new cases, nothing got worse. Here two inputs changed: the model, and the weekly rebuild of the help-centre index. Four runs separate them. New model on last week's index scores 184; old model on this week's index scores 178. The model is not the cause. The index is.
Where? Cut the 10 flips by failure class first, because the classes carry different rules. Eight are policy tickets; two of them are wrong on a fact. Read those two: the rebuild dropped the annual-plan refund article after its URL changed, so drafts fell back to the 14-day window — the failure from module 0, returned through a different door.
When to stop diagnosing
Stop when the next cut cannot change the action. Here that point comes in two halves. The four runs decided what to revert: the index, not the model. The class cut decided when: two new factual failures are 1% of 200 on their own, against a 0.5% blocking rate, so the build is blocked today. After that, whether the eight policy flips sit in one tenant or five changes nothing. Restore last week's index, re-run to confirm 184, keep the model update, and add a check to the rebuild that fails when the article count drops.
Three conditions trigger the stop: a blocking class has fired; you can name the one change to revert and the version before it passed; or every remaining cut leads to the same action. Price the cost of not stopping. If edit-before-send rose from 14% to 17% that week, 3 points × 200,000 drafts is 6,000 more edited drafts a day; at 20 seconds each that is 33 agent-hours, about $1,000 a day at an assumed $30 an hour, before any wrong refund window an agent missed.
The opposite mistake costs as much. A candidate who rolls back the model because it was the announced change keeps the broken index, loses whatever the update improved, and watches Monday's run read 178 again. The experiment lesson asks for the pre-registered rule before the readout; this round asks for the same thing in a different form: the blocking classes were written into the PRD before any update arrived, so the stopping rule existed before the data did.
In an interview
What is being tested is whether you can turn a moved number into one decision inside the round. Say the four levels before running any, give each as one query with the result you expect, and announce the stop: "two new factual failures are 1% on their own against a 0.5% blocking rate, and the four runs say it is the index, so I restore the index today and diagnose the rest on the old one." That sentence scores order, instrument and stopping at once.
The mistake that loses points is a diagnosis with no stopping rule — query after query until the interviewer calls time, and nothing on the sheet. The second is reverting the change that was announced rather than the change that was found.
Check yourself
1. The North Star fell 10%; tickets fell 10% and the other three inputs are flat. What do you change this week?
Nothing in the product. 2,250,000 × 0.60 × 0.40 × 0.90 = 486,000, so the fall is volume, which belongs to the tenants' businesses. Check the calendar for a holiday in a large tenant's region and say so in the readout; a fix for a fall the product did not cause is a change shipped against noise.
2. A model update cuts the per-call price by 30%. The run falls to 88%, and all 8 net flips are tone, no blocking class. Roll back or keep?
Roll back now and come forward deliberately. 88% is under the 92% done line whatever the class, and the last passing build is one config change away. The update is worth about 30% of $54,000, roughly $16,000 a month, so fix tone in the prompt on the new model in a branch and switch when that pair scores 92% with no blocking failure.
3. After restoring the index, the re-run scores 183, not 184. Does the fix stand?
Yes. One case is half a point, inside the margin, and the fact class is back to last week's count, inside the gate that fired. Read the one flip; if it is a borderline tone case that flips between runs, mark it unstable and move on.