Runbooks and the handover doc
Write runbook entries a stranger can follow at 02:00, assemble the handover doc, and get each sign-off for something the customer did without you.
On this page
The person who reads your runbook was in none of your meetings: someone on the customer's IT rota, paged at 02:10 by an alert for a service they have never seen, who will follow the entry literally because at 02:10 that is all anyone can do. The runbook and the handover doc are written for that reader, and the handover meeting proves, before you leave, that the reader can use them without you.
What the on-call needs at 02:00
The SRE book (Beyer, Jones, Petoff and Murphy, Site Reliability Engineering) sets the conventions here. A page is for something that needs a human now and needs judgement; anything a script could do, the service does itself. Every alert that can page has an entry, and the entry gives mitigation before diagnosis: make the system as safe as it can be, then find out why. The book's introduction reports that recording the response in a playbook ahead of time gives roughly a threefold improvement in mean time to repair over improvising.
The entry answers the on-call's questions in the order they ask them:
- Does this need me now? Claims arrive from 08:00 to 18:00, and when the tool fails, handlers get the old search box, so almost nothing that breaks at 02:00 needs fixing before 07:30.
- What is the claims floor seeing right now?
- What do I do first to make it safe?
- How do I find out why, in which panel?
- Whom do I call, by role and rota?
- What must I not do?
A monitoring rule follows from the first question: an alert whose honest answer is "not until 07:30" opens a ticket and wakes nobody.
The mistake: a design doc titled "runbook"
The runbook most engineers write in week 7 is the architecture in their heads, written down: four pages on the components, the data flow from claims.fnol_v through the index to pilot.triage_output, the prompt's structure, the retry policy. All of it is correct, and none of it answers question one. The alert the on-call received is not named; "fallback" appears on page three, in a paragraph about design; there is no command anyone could paste. It is a document for the next engineer, and the on-call needs one for 02:10 tonight.
One entry, written for 02:00
The pilot has four alerts, and each gets an entry in the same shape. The one for drafts stopping:
ALERT triage-drafts-stalled
PAGES 07:30–18:00 only; outside those hours it opens a ticket
MEANS no triage record written for 10 minutes while claims are arriving
IMPACT handlers get an empty draft and the old search box; no claim is lost
FIRST dashboard "claims-triage", panel "fallback": empty drafts served should be rising
if it is flat, claims are waiting on the pilot: set flag triage.enabled = false
THEN panel "model calls":
5xx from the model endpoint -> vendor status page; wait; when it recovers,
run job triage-rerun for the ids logged since the alert
401 invalid_client -> the identity was disabled: entry 3; do not restart
CALL pilot owner, claims QA lead, rota "claims-ai", working hours only
NEVER restart in a loop; re-enable a disabled identity; edit the prompt
PAGES answers question one first. IMPACT says how worried to be: the fallback in where the model sits means the floor works as it did before the pilot. FIRST is the mitigation, with one flag that forces the fallback if it did not engage. THEN branches, each branch ending in an action. CALL is a rota, because a person's name is wrong the day that person leaves. NEVER lists what turns a degraded morning into an outage: restarting against a disabled identity is a retry loop against the customer's off switch, which auth and the boundary says the service must obey.
The other three entries are shorter. Nightly index build failed: a ticket, never a page, because yesterday's index keeps serving. Identity refused: do not restart or re-enable; check the identity platform's change log; ticket to the IAM lead's queue. Rollback trigger fired: claims operations throw the per-segment switch from crawl, walk, run without waiting, and the alert pages the IT on-call, who checks last night's index build for a wording never re-indexed. A drop in quality is not an alert; it is the owner's eval reading, after each wording issue and in the monthly drift check.
The page budget is one line of arithmetic. Assume the endpoint stalls drafts twice a month, the rollback trigger, re-derived at five a day for all four teams' motor and property claims, fires by chance once in six months, and the identity is refused once a quarter: about 2.5 alerts a month, paging only between 07:30 and 18:00. The SRE book budgets at most two incidents per 12-hour shift, so the risk is not overload but a page so rare that nobody remembers the runbook, which is why the drill below is repeated quarterly, with the IT on-call lead as R.
The handover doc
One page, with attachments, read by the people in the RACI and kept by the owner:
- What the system does, in two sentences, and what it does not do.
- The five lines of the definition of done with their week-8 readings, and who reads each from now on.
- The RACI from week 9, with no R or A left in the vendor's column.
- The runbook: four entries.
- The credential list, with the service identity's owner of record moved to the customer's IAM lead and the model API key rotated, so the key you knew is not the key that runs.
- The one-page threat model the security lead signed.
- Open issues, each with an owner and a date: commercial's walk decision, the run gate for small motor claims.
- How to re-run the eval: one command, and where the 200 cases live.
- The route to the vendor: the support queue and the contract reference, not your phone number.
The handover meeting and the sign-off
The sign-off is three signatures, each given for something the customer did without you, not for something you showed them.
The drill comes first. On Tuesday of week 7 the IT on-call lead runs the drafts-stalled entry against a staged failure, the model endpoint blocked at the firewall for 45 minutes, while you sit in the room and say nothing. Every point at which they had to ask you is a missing line; it is written and the drill re-run on Thursday. Then the QA lead re-runs the eval from the doc's one command, on her own machine, and gets the week-7 reading; if she cannot, the eval is not handed over. Then the credentials move, and the customer's IAM lead re-runs the revocation test the credential list describes.
The meeting takes 30 minutes on Friday of week 8. The five lines are read with their numbers. The RACI is read row by row, and each person holding an A says yes out loud, which is when an empty cell surfaces. Open issues get owners and dates. The VP signs.
The first week after you leave
You are consulted, not responsible: two half-hour office hours a week for two weeks, booked through the vendor's support queue so the customer learns the route they will use in month six. Your access ends at week 8 plus 14 days and is not extended "just in case"; standing access to a customer's claims is a finding on their next security review.
Measure the handover by how many times someone needed you. The target is zero, and each call is a missing line in the runbook or a missing name in the RACI, fixed that day. The first wording change after handover is the real test: if the owner re-indexes, re-runs the eval and approves the change without you, the deployment is theirs.
In an interview
The deep-dive and the behavioural round both ask how you hand over. The interviewer listens for a runbook tested by someone other than its author, alerts that know whether they can wait for morning, a rotated key, and an end date on your own access. Describe the drill: "the on-call lead ran the entry against a staged failure while I said nothing; two steps needed me, and we rewrote both before Thursday."
The mistake that loses points is a handover that is a presentation: "we walked them through the system". The interviewer asks what the customer did without you before you left, and if the answer is nothing, the first incident is the drill.
Check yourself
1. In the Tuesday drill the on-call lead takes 22 minutes to find the fallback panel. Handover is Friday of week 8. Do you sign off on schedule?
Not on that entry. Fix the entry, not the person: put the dashboard link and panel name in the FIRST line and re-drill on Thursday. If Thursday runs clean, sign on schedule; if not, move the sign-off a week, which costs less than an on-call who spends 22 minutes of a real failure not knowing whether claims are waiting.
2. The IT platform's default rule pages on any credential refusal, at any hour. Should the pilot's identity-refused alert follow it? Estimate what waking someone buys.
No. A refusal is the customer's off switch, deliberate or accidental, and the IAM lead who can reverse it works office hours. Waking the on-call at 02:00 buys nothing before claims start at 08:00, since the fallback serves empty drafts either way. Make it a ticket overnight and a page from 07:30, so the day shift chases the IAM lead before the 09:00 peak.