Metrics and analytics · Analytics

Cohorts and funnels

Read a retention grid along its diagonals, see why the blended average hides the newest cohort, and work a 60% drop at one funnel step down to the segment that caused it.

Lesson · 7 min · 21 of 59

On this page

A cohort grid is the retention query's result laid out so the eye can read it: one row per sign-up week, one column per week since sign-up, each cell the share of the cohort still active. Read down a column to see whether the product is getting better for new users; read along a row to see how one group ages. A funnel is the other instrument, and a 60% drop at one step is the most common readout a PM gets; what to check first is the second half of this lesson.

Reading the grid

Six weekly cohorts of the self-serve tier, each measured on the share that sent a draft in week N after their first one:

Down a column is the product changing; along a row is one cohort ageing; the blank corner is time that has not passed yet.

Three readings. Along the 3 August row: 100, 62, 51, 46, 44, 43 — the curve flattens by week 3, which means the users who stay past three weeks mostly keep staying. That flattening is the number to care about; a curve that keeps falling is a product people try and leave, and no acquisition spend fixes it. Down the week-1 column: 62, 60, 58, then 64 and 70. An onboarding change shipped on 24 August, and the two cohorts after it retain better at week 1 than the three before. Two cohorts is thin evidence, and the experiment lesson says how thin; but the grid is where the signal shows first, and it shows it by column. The blank corner: the 31 August cohort has no week-2 figure because week 2 has not happened for them yet. Blank is not zero.

The grid is the retention query pivoted, with one addition: the cell is reported only once every member of the cohort has had the whole week to act. A week-2 cell for a cohort whose last member signed up five days ago is a partial week for that member, and a partial week understates retention for the whole cohort.

The mistake: the blended average

A dashboard tile labelled "week-1 retention" that averages across all cohorts reads 62.8% — (62 + 60 + 58 + 64 + 70) ÷ 5 — and has read within a point of that for a month. The onboarding change lifted week-1 retention from about 60% to about 67% on the two cohorts that got it, and the tile cannot show it, because the three older cohorts at 60% outnumber the two newer ones and will do so until the old cohorts age out of whatever window the tile uses. The average hides the newest cohort, which is the cohort the latest change affected, which is the one the team needs to see.

Worse is the tile that averages across all users rather than all cohorts, so the 3 August cohort, now in week 5 at 43%, pulls down a "retention" figure that the 31 August cohort at 70% in week 1 cannot pull up. That tile falls every week as the product grows, because every new cohort enters at week 0 and the old ones keep ageing; a steady product looks like it is decaying.

The fix is to never show retention as one number. Show the column for one fixed week, by cohort, newest cohort last, so the latest change is the rightmost point. "Week-1 retention, by sign-up week" is a line that moves when the product does.

A funnel where step 3 drops 60%

The in-product funnel for one tenant's agents over a week: 10,000 tickets opened; 6,000 received a draft (coverage, 60%); 5,400 drafts were opened by the agent (90%); 2,160 were sent without an edit (40%); 1,944 had no reopen in 7 days (90%).

Step 3 loses 60% of what reaches it. The question is not why agents edit; it is which drafts they edit.

Step 3, draft opened to sent unchanged, converts at 40%: 3,240 drafts a week are edited or discarded. A 60% loss at one step is the readout that starts a roadmap argument, and the argument is usually about the wrong thing, because the first instinct is to explain the step ("agents do not trust the model") rather than to segment it.

What to check first, in order:

Is the step measured? draft_sent fires when the agent clicks send in the web client. If the tenant's agents use the desktop client and it logs draft_sent as reply_sent, the step is undercounted, not failing. Twenty minutes with the events table rules this out, and it is the cause more often than anyone admits.
Segment by what is nearest the step. The thing the agent sees is the draft, so split by draft length. Drafts under 120 words: 55% sent unchanged. Drafts over 120 words: 18%. The 60% loss is a length problem concentrated in two fifths of the drafts (0.6 × 55 + 0.4 × 18 ≈ 40), and the fix is the output cap the cost lesson already wanted for its own reasons.
Then by time. Was acceptance 40% last month too? A step that was at 55% and fell to 40% in one week has a release behind it; a step that has always been at 40% is a design fact.

Only after those three does the qualitative question — why do agents edit — earn an interview, and by then the interview has a specific prompt: "show me the last long draft you rewrote".

One point at step 3 is worth

The product form from the North Star lesson prices the step: for this tenant, 6,000 × 0.90 × 0.40 × 0.90 = 1,944 untouched resolutions a week. Lifting step 3 from 40% to 50% by capping draft length: 6,000 × 0.90 × 0.50 × 0.90 = 2,430, so 486 more resolutions a week for one tenant, from a prompt change. Across all tenants, 2.5 million tickets a week, the same lift is worth 2,500,000 × 0.60 × 0.10 × 0.90 = 135,000 a week, a quarter again on the North Star's 540,000.

That is the number the readout should end with. "Step 3 drops 60%" is a symptom; "long drafts convert at 18%, capping length is worth about 135,000 resolutions a week" is a decision.

In an interview

Execution questions come as a grid or a funnel: "retention is falling" or "conversion at step 3 dropped". Read the grid by column before answering, and say which column: "week-1 retention by cohort is flat at 60 until the 24 August cohort, so whatever changed, changed then". For the funnel, say the three checks in order — instrumentation, the nearest segment, time — and name the segment you would cut first and why it is nearest the step.

The mistake that loses points is explaining the drop before segmenting it — "agents do not trust AI drafts" is a story, 18% on long drafts is a finding — and reading a blended retention average as if it were a cohort. An interviewer who hears "retention is 62.8%" will ask "for which cohort, at which week?", and the candidate who cannot answer has not read the grid.

Check yourself

1. The week-1 column reads 62, 60, 58, 64, 70 and the change shipped before the fourth cohort. Do you declare the onboarding change a success?

Not yet. Two cohorts of about 1,000 each at 64 and 70 against three at about 60 is suggestive; the lift is 7 points on a base of 60, large enough that two more cohorts at 65 or better would settle it. Say "two cohorts up, two more to confirm", and hold the launch announcement a fortnight. What would reverse it: the next cohort at 59.

2. Step 3 is 40% overall, 55% for short drafts and 18% for long ones, and long drafts are two fifths of the volume. Where does capping length move the overall step?

Three fifths at 55% and two fifths at 18% gives 0.6 × 55 + 0.4 × 18 = 40.2%, the reported 40%. If capping brings long drafts to 45%, the blend moves to 0.6 × 55 + 0.4 × 45 = 51%. Promise 50%, measure on next week's cohort of tickets, and check the no-reopen step did not fall, since a shorter draft that resolves less is not a win.

3. A tenant's dashboard shows "retention 38%" and the tenant is alarmed. What is the first question?

"Which cohort and which week?" If it is the all-users blend, it falls as they grow and means nothing; ask for week-1 by sign-up week. If week-1 by cohort reads 38% for the newest cohort against 60% for the ones before, that is real and the next question is what changed the week that cohort joined.