Brand stewardship of generated output
Write a review standard for generated output, gate every paid customer-facing piece, sample 5% of the rest, and catch drift monthly on a fixed set.
On this page
When anyone on the team can produce fifty pieces of copy before lunch, the brand is whatever the worst of them says. Brand stewardship keeps the worst piece inside a standard: a written review standard, a rule for which output is reviewed before it ships and which is sampled after, and a monthly check that catches the output changing when nobody changed anything. For the running company it costs about 30 hours a month, and the arithmetic says where each hour goes.
The mistake: everyone approves their own
The default is that whoever generates a piece reads it and ships it. It feels like review, and it measures nothing. Each person sees only their own output, so nobody knows the defect rate. And a defect the author does not see as one, "the fastest way to get paid" in an ad, last year's price in an email, passes every time it is generated, because the same reader misses it every time.
The invoicing app, at $3M ARR with a marketing team of three, generates about 1,450 pieces a month:
| Output | A month | Paid and customer-facing |
|---|---|---|
| Ad variants | 240 | yes |
| Landing-page variants | 10 | yes |
| Lifecycle email variants | 160 | no |
| Social posts | 120 | no |
| Call and interview summaries | 400 | no, internal |
| Page titles and descriptions | 300 | no |
| Creative versions for organic channels | 220 | no |
The fix has three parts: a standard that says what a defect is, a rule for where review happens, and a check that the machine has not moved.
The review standard: six checks
A standard a reviewer can apply in four minutes is a short list of pass-or-fail checks, ordered by what a failure costs.
- Claims. Every claim is one the product can prove, with the source on file. The positioning work in Dunford's method decided what the company claims; generated copy drifts towards claims nobody decided.
- Facts. Prices, plan names, feature names and dates are current.
- People and disclosure. No quote, review or person is presented as a customer unless real and on file, and every disclosure line the campaign needs is present, as ethics and disclosure sets out.
- Voice. The brand's voice rules, written so they can be tested: no superlatives, numbers before adjectives, no exclamation marks.
- Audience. The piece matches the segment and the lifecycle stage it was written for.
- Channel. Length, format, the link and its tracking tag.
Checks 1 to 3 are severe: a failure can mislead a customer or break a rule. Checks 4 to 6 are minor: fix it and count it. The split matters because it decides what a single failure triggers, below.
Gate paid output, sample the rest
The rule is 100% review of paid, customer-facing output before it ships, and a random 5% of everything else after it ships.
Why all of the paid output. Spend multiplies the reach of each piece. One ad variant with $1,000 behind it, at the course's $20 CPM, is shown 50,000 times, so a wrong claim reaches 50,000 people before the weekly report notices. Paid output also carries the claims and endorsements the advertising rules cover. A four-minute review costs $6.67 at $100 an hour. Sampling would not do: at a 5% defect rate, 240 ads a month include about 12 bad ones, and a 5% sample reads 12 ads and finds 0.6 of them, so about 11 defective ads a month would run with money behind them.
Why a random 5% of the rest. Everything else (summaries, page titles, organic posts, email variants) is read by its author before it ships. The sample is not a second review; it measures how good the first one is. It has to be random, drawn by a script, because a sample the author picks ("the ones I wasn't sure about") measures their doubts, not the rate.
The arithmetic of one month, 60 pieces from 1,200, with 3 defects found:
defect rate 3 / 60 = 5%
95% interval 1.96 x sqrt(0.05 x 0.95 / 60) = ±5.5 points, so 0% to about 10.5%
three months 180 pieces, same rate = ±3.2 points
One month's sample can tell 5% from 15%; it cannot tell 5% from 8%. That is enough for a monthly alarm, and the quarter's pooled 180 pieces give the number worth reporting. A severe defect is different: one found in the sample means a template is producing it, so that template goes to 100% review until it has two clean months.
The hours: 250 paid pieces at 4 minutes is about 17 hours, 60 sampled pieces at 6 minutes is 6, the drift check below is 4, and writing the monthly note about 3. Thirty hours, $3,000 a month, is the figure the task ledger carried.
Drift: a fixed reference set, scored monthly
The sample says how often shipped output missed the standard. It cannot say why the rate moved, because the briefs change every month. Drift needs a fixed input.
The reference set is 40 briefs, written once and never edited: 10 ads, 10 emails, 10 social posts, 10 page snippets. Each month the steward runs all 40 through the current setup (the prompt templates, the examples in them, and whatever model the tool uses that month) and scores each output on the six checks: 240 results. In the baseline month, 228 pass, 95%.
Generated output varies from run to run with nothing changed, because generation samples from a distribution (sampling and determinism explains why). So in the baseline month the steward runs the set twice; if the runs differ by 6 checks, a month-on-month change smaller than 6 is noise. The alarm sits at 216, twice the noise below the baseline. This is the marketing team's version of an evaluation set: the same inputs and the same rubric every time, so a moved score means the setup moved.
What moves it, in practice:
- The model behind the tool changes. A vendor updates the model under the same product name; outputs get longer, or superlatives creep back.
- A template is edited. Someone improves one prompt for one campaign and it becomes the default.
- The examples go stale. The sample copy inside a template quotes last year's price.
Worked: in month 5 the score falls to 204 of 240, 85%. Of the 36 failures, 20 are check 4, superlatives, all in ads and social posts, and the tool's changelog shows a model update in week 2. The steward adds two voice examples to the ad template, re-runs the set (226), and the shipped-output sample for the month confirms the rise in voice defects. Without the reference set, the sample would have shown voice defects rising and nobody would have known whether to blame the briefs, the people or the tool.
Sampling and thresholds for an automated reviewer are worked in sampling review and confidence thresholds; the same arithmetic applies when the reviewer is a person.
In practice
To the team: one standard, six checks, one named steward, and no paid piece ships without the steward's pass. To the founder: "Thirty hours a month, $3,000, tells us our defect rate within a few points and catches a model change within a month." To a tool vendor: can the model behind your product change without notice, and how would we know? The reference set is the answer if theirs is "it improves continuously".
The mistake that costs money is sampling paid output instead of gating it. At a 5% defect rate, a 5% sample of 240 ads a month lets about 11 defective ads run with spend behind them. Reviewing all 240 costs 17 hours.
Check yourself
1. This month's sample of 60 finds 6 defects, 10%, against 3 last month. None is severe. Do you act?
Not on the rate alone. 10% on 60 pieces is ±7.6 points, roughly 2% to 18%, which overlaps last month's 0% to 10.5%. Check the drift score: if it held, the briefs or the people changed, not the setup. Sample 10% next month, 120 pieces, to tighten the reading. Had any of the 6 been severe, the template behind it would go to 100% review now.
2. A creative test will raise ad variants from 240 to 1,000 a month: 20 headlines combined with 50 images. Do you keep 100% review?
Keep 100% of the components, not the combinations. 1,000 variants at 4 minutes is 67 hours, more than a third of a person's 176. The 20 headlines and 50 images are 70 reviews, under 5 hours, and the combinations are assembled by rule from approved parts, with the campaign-QA workflow checking the assembly. What would change it: a headline whose meaning depends on the image it sits with, which goes back to whole-ad review.
3. The tool's vendor says a new model replaces the current one next week. What do you do, and what does it cost?
Run the reference set on both before the switch: 80 outputs at 6 minutes is 8 hours, $800. Accept the new model if it scores at least 222, within the noise of the 228 baseline; if it scores lower, fix the templates against the failing checks before it becomes the default, or ask the vendor to pin the old version until you have.