Rollout, handover and writing · Handover

The runbook nobody could run

Find why a runbook entry turned a disabled credential into 83 minutes and a security incident, cost the morning, and fix your own handover before it ships.

critiquehardExercise · 3 min · 56 of 64

On this page

An on-call engineer followed a runbook entry to the letter and made the morning worse. The defects are in the entry and in the handover behind it, and your own runbook is due today.

The situation

Monday of week 7 of your claims-triage pilot, the day your runbook is due. A colleague handed over the same product at another regional insurer three weeks ago: 1,200 claims a day, 240 an hour in the 09:00–12:00 peak, and the same empty-draft fallback. Last Tuesday at 09:12 its alert triage-errors-high paged an IT on-call who had joined the rota after the handover. The entry, written in week 7 and never drilled, is below. The on-call restarted the service four times by 09:40; each restart retried the token request three times, the identity platform flagged the twelve failures as credential stuffing, and the security lead opened an incident. Your colleague's phone went to voicemail; her access had ended the week before. At 10:35 the IAM lead found the cause: the quarterly access review had disabled the service identity, whose owner of record was still your colleague. Handlers had empty drafts and the old search box throughout, and no claim was lost. Assume the panel saves 3.3 minutes on each motor and property claim, 80% of volume.

triage-errors-high
Triage service errors are above threshold.
1. Check the pilot logs for errors (namespace claims-ai).
2. If the model API is returning errors, restart triage-svc.
3. If errors persist after restart, contact Priya (FDE); mobile in the team wiki.
4. Escalate to the vendor if needed.
See the architecture doc (v3, 14 pages) for the data flow.

The question

Find the defects that turned a disabled credential into 83 minutes and a security incident, rank them by what each cost, and say what your own entry and handover will do differently.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

The on-call did what the entry said; the entry failed. Ranked by cost:

Step 2 restarts on any model error. A 401 is the customer's off switch, and restarting against it is a retry loop: twelve token requests, a security incident, and an IAM team investigating instead of re-enabling. It needs a branch per error: a 5xx waits for the vendor and re-runs the logged claims; a 401 never restarts and goes to the IAM lead's queue.
Nothing says whether this needs anyone now, or what the floor sees. The fallback served empty drafts all morning, so the floor was degraded, not down; an impact line and a named fallback panel would have stopped the restarts at 09:15.
The call line is a person whose access had ended; it should be a rota, with the vendor's support queue behind it.
Step 1 is diagnosis before mitigation, and "the logs" names no panel; the architecture doc is for the next engineer.

Behind the entry is the handover: the identity's owner was never moved, so the access review did its job, and nobody drilled the entry; a staged 401 would have exposed every line above in fifteen minutes.

The cost: 09:12 to 10:35 is 83 minutes at 240 claims an hour, about 332 claims, 266 of them motor and property; at 3.3 minutes each, about 15 handler-hours, plus the incident. Written in the six-question shape of runbooks and the handover doc, the same morning is one IAM ticket.

Your own handover: move the identity's owner this week, write the entry in that shape, and drill it on Tuesday with you silent.

Rubric

places the failure in the entry and the handover, not in the on-call who followed it
ranks the restart step first, and says why restarting against a 401 made the morning worse
adds a does-this-need-me and impact line tied to the fallback that was already serving
replaces the named engineer with a rota and the vendor's support queue
traces the root cause to an identity owner never moved and an entry never drilled
costs the morning from the stated constants: 83 minutes, about 332 claims, about 15 handler-hours

Where answers go wrong

"The on-call should not have restarted four times; retrain the rota." Loses because the entry told them to restart, and the next new member of the rota will follow it the same way; the fix is the line, not the person.

"Add detail: link the architecture doc and the full log schema." Loses because the entry already pointed at fourteen pages; length was not the defect, order was: impact first, mitigation before diagnosis, one branch per error.

"The access review broke it; exempt the pilot's identity from reviews." Loses because the review did its job, since an identity owned by someone who has left should be disabled; the defect is a handover that left the owner unchanged, and an exemption is a finding on the next security review.