Estimate the eval budget for a launch

A strong answer

Size the set from the gate. At p = 0.9, 200 items give √(0.09 ÷ 200) ≈ 2.1 points of standard error, so a 3-point drop is 1.4 standard errors and the set cannot tell it from noise. For the drop to be two standard errors, √(0.09 ÷ n) ≤ 1.5 points, so n ≥ 0.09 ÷ 0.015² = 400. Two hundred is not enough; 400 is.

People: 400 × 60 s ≈ 6.7 hours, 8 with the second pass; judge calibration 50 × 3 minutes = 2.5 hours. About 10.5 hours, $420 — under two working days of one person's time.

Tokens for one run over 400 tickets:

feature  400 × 9.1k = 3.64M in  × $3/M  = $10.92
         400 × 360  = 144k out  × $15/M = $2.16
judge    400 × 4k   = 1.6M in   × $3/M  = $4.80
         400 × 100  = 40k out   × $15/M = $0.60
one run                                 ≈ $18.50

Runs: (10 revisions + baseline) × 3 = 33, about $610. The budget is about $1,030: $610 of tokens and $420 of people.

The money is small; the ordering is the constraint. The 400 labels must exist before the first revision is scored, so labelling starts this week, owned by the support lead rather than the engineer writing the prompt.

Reversal: if product accepts a 4-point gate, √(0.09 ÷ n) ≤ 2 points gives n ≈ 225, about half the labelling and the tokens. Comparing two unpaired totals would need 800, since each total carries its own error; the paired diff in regression gates cancels every item that does not flip, which is what makes 400 enough.

Rubric

• sizes the set from the gate with the standard-error arithmetic shown, and rejects 200
• converts items to labelling hours, including the second pass and the judge calibration
• costs one run with feature and judge tokens, input and output separately
• multiplies by revisions × repeats plus the baseline, and states one total
• names the ordering constraint on the labels and who labels
• says how the set size changes if the gate loosens

Where answers go wrong

"Two hundred items, because that is the usual golden-set size." A size copied, not derived. At 2.1 points of standard error the gate it was bought for cannot reliably catch the 3-point drop it exists to catch — the eval-it family in what an interviewer asks an AI engineer turns on exactly this.

"An eval run costs a few dollars, so the budget is negligible." It prices one run of the feature, forgets the judge and the 33 runs, and leaves out the 10.5 hours of people — the one line that has to be scheduled.

"Run each revision once to save money." It saves about $400 and removes the only measure of run-to-run noise, so a 3-point move can no longer be told from sampling.