North Star, input and output metrics
Write a North Star as a product of three or four inputs, compute which input is worth a point this week, and keep guardrails out of the formula.
On this page
A North Star is one output number the business cares about, and nobody on the team can move it directly. What the team can move is three or four inputs that multiply into it. The work of this lesson is writing the multiplication down, because once it is written the question "what do we do this week" has an arithmetic answer rather than a debate.
One output, and what it is not
The drafting assistant this course follows sits in front of 2.5 million support tickets a week across every tenant; at the 0.60 coverage below, that is 1.5 million drafts a week, about 210,000 a day, the scale of the 200,000 drafts a day the token bill prices. Its North Star is resolutions without a human touch per week: tickets closed by a reply nobody edited, whether the agent approved it with one click or the tenant let it go out on its own. It is an output because a customer and a finance team both recognise it as value: a resolved ticket that cost no agent minutes.
Two candidates lose to it. Revenue from the add-on is the real goal, but it lags the product by a renewal cycle, so a quarter's work shows up a year later and the team cannot steer on it. "Drafts generated" is immediate and moves every time an engineer ships, which is exactly the problem: the model can generate 1.5 million drafts a week while resolving nothing. A metric that goes up when the product gets worse is a vanity metric, and the test for one is simple: can it rise while the customer's outcome falls?
The product form
The output is a product of inputs the team owns one each:
resolutions without a human touch
= weekly tickets × intent coverage × draft acceptance rate × 7-day no-reopen rate
= 2,500,000 × 0.60 × 0.40 × 0.90
= 540,000 a week
Intent coverage is the share of tickets the assistant recognises well enough to draft for (0.60: the other 40% fall outside the intents it was trained and evaluated on). Draft acceptance is the share of those drafts sent without an edit (0.40). The 7-day no-reopen rate is the share of sent replies the customer did not come back on within a week (0.90); a reopen means the resolution was not one. Weekly tickets is the fourth term and the one the team does not own; it belongs to the tenants' businesses.
The form matters more than the names. Because the output is a product, a relative change in any input is the same relative change in the output: lift coverage by a sixth and the North Star rises by a sixth. That is what turns "which input do we work on" into a comparison of costs.
Which input to move this week
Three proposals arrive in the same planning meeting. Each is a lift to one input; the arithmetic ranks them.
Coverage from 0.60 to 0.70, by training the classifier on two more intents: 2,500,000 × 0.70 × 0.40 × 0.90 = 630,000, so +90,000 a week. Acceptance from 0.40 to 0.45, by fixing the tone failures the eval flags: 2,500,000 × 0.60 × 0.45 × 0.90 = 607,500, so +67,500 a week. No-reopen from 0.90 to 0.95, by refusing to draft when retrieval confidence is low: 2,500,000 × 0.60 × 0.40 × 0.95 = 570,000, so +30,000 a week.
The ranking is not the decision on its own; it is half of it. Coverage is worth 90,000 and costs two intents' worth of labelled data and an eval extension, perhaps three weeks. Acceptance is worth 67,500 and might be a prompt change, a week. 67,500 in one week beats 90,000 in three. What the form gives you is the numerator; RICE supplies the rest.
Notice what the no-reopen proposal does: refusing to draft at low confidence raises the no-reopen rate by lowering coverage. If refusals take coverage from 0.60 to 0.55, the output is 2,500,000 × 0.55 × 0.40 × 0.95 = 522,500, a loss of 17,500 a week. A proposal that moves two inputs in opposite directions has to be scored on the product, never on the input it was named for.
The mistake: optimising a ratio by shrinking its denominator
A team told to raise draft acceptance will find the fastest way: draft only for the easy tickets. Coverage drops to 0.45 and acceptance climbs to 0.50, and the dashboard tile for acceptance turns green. The output: 2,500,000 × 0.45 × 0.50 × 0.90 = 506,250, down 33,750 from where it started.
The fix is structural. Every input gets reviewed beside the output in the same weekly readout, and no team owns a ratio without also being accountable for the count under it. The product form makes the trade visible before it happens; a sum of weighted metrics, the common alternative, hides it, because a sum lets one term rise while another falls and still reports progress.
Guardrails are not inputs
First-response time does not appear in the formula. A faster first response does not multiply into more untouched resolutions; a slower one does not reduce them. It is a constraint the product must not break: a guardrail. The same goes for cost per resolution, which comes from the token bill, the complaint rate, and p95 latency, which is what the agent feels rather than the mean (the tail-latency lesson explains why).
The difference in handling: an input is something you push; a guardrail is something you watch, with a threshold written down before launch and a response written beside it. The AARRR lesson sets the thresholds. The test for which is which is the one above: does it multiply in?
Reading the number weekly
540,000 a week is a baseline, not a target. The readout the team sees every Monday is the four inputs and the product, this week against last, and one line saying which input moved and why. When the product rises and no input the team touched did, the weekly ticket count did it and the team should not take credit; when an input the team shipped against moved and the product did not, another input absorbed it, and that is the thing to find. The SQL lesson writes the three queries that produce the row.
In an interview
The execution question is "the metric dropped 10%; what do you do?" and the product-sense question is "what metric would you pick for this feature?". Both are testing whether you can write the product form. Answer the second one by naming the output, naming the three or four inputs, saying which one the team owns most directly, and then naming one guardrail and explaining why it is not an input. Answer the first by segmenting along the inputs: a 10% drop in the output is some combination of drops in four terms, and the first query tells you which.
The mistake that loses points is naming a metric that rises when the product gets worse — drafts generated, assistant sessions, "AI usage" — or naming a ratio with no count beside it. An interviewer who hears "acceptance rate" asks "acceptance of what, out of how many?" and the answer has to be ready.
Check yourself
1. The weekly ticket count grows from 2.5 million to 3 million and the output rises to 648,000. Did the product improve?
3,000,000 × 0.60 × 0.40 × 0.90 = 648,000 exactly, so all three product inputs are unchanged and the growth is volume. The readout should say so. Credit for this week goes to the tenants' businesses, and the team's own question — which input moved — has the answer "none".
2. Engineering can deliver either coverage to 0.65 in two weeks or acceptance to 0.44 in one week. Which ships first?
Coverage to 0.65: 2,500,000 × 0.65 × 0.40 × 0.90 = 585,000, +45,000 for two weeks. Acceptance to 0.44: 2,500,000 × 0.60 × 0.44 × 0.90 = 594,000, +54,000 for one week. Acceptance first: more output per week of effort, and it lands earlier. Even at one week, coverage's 45,000 would still lose to 54,000. The evidence that would reverse it is an acceptance estimate that slips to three weeks: 54,000 ÷ 3 = 18,000 a week of effort against coverage's 45,000 ÷ 2 = 22,500.
3. A tenant asks for a dashboard tile showing "AI replies sent". Do you add it?
Only beside the reopen rate for the same replies. Replies sent can rise while resolutions fall, so alone it is a vanity number, and a tenant who steers on it will turn auto-send on for tickets it should not cover. Pair it with its denominator and its guardrail, or decline.