The role, its neighbours and the interview · The interview

The case round — a triage pilot in 45 minutes

In a 45-minute case round, find where a facilities firm's missed emergency deadlines come from and decide what its pilot builds first.

casehardExercise · 4 min · 61 of 64

On this page

The interviewer hands you a customer you have never seen and a promise somebody else made. The round is scored on whether you find where the time goes before you design anything, as the typical loop describes.

The situation

The interviewer plays the COO of a facilities-management firm and gives you 45 minutes. The firm receives 2,400 maintenance requests a working day by email and portal, between 08:00 and 18:00, half of them between 08:00 and 11:00. Twenty-four dispatchers work one shared queue, first in first out, at a median 6 minutes a request: 1 minute reading it, 4 minutes finding the client's service contract among 180 PDFs to choose the contractor and response tier, 1 minute raising the work order. About 8% of requests are emergencies, such as a burst pipe or a power loss, which the contracts require dispatched within 30 minutes. Last quarter 41% of emergencies missed that, and the penalty credits are why the COO is in the room. The deal promises "every emergency dispatched within 30 minutes" after an 8-week pilot. The COO opens: "We want AI to do the dispatching." Assume 0.75 words per token, input at r per million tokens and output at 4r, and about 750 words of request plus instructions per call.

The question

Decide what the pilot builds first and what definition of done you ask the COO to sign, with the arithmetic you would put on the whiteboard.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Locate the minutes first. Capacity is 24 × 60 ÷ 6 = 240 requests an hour. The morning brings 400 an hour, so the queue grows 160 an hour; an emergency arriving at 10:00 waits behind about 320 requests, 320 ÷ 240 ≈ 80 minutes. The misses are queue position, not handling time. Strict first in first out would miss about 80%, since demand fills all 24 × 600 dispatcher-minutes; dispatchers already pull some forward by eye.

So the first build flags emergencies on arrival and moves them to the head of the queue. A dispatcher frees up every 15 seconds (6 minutes ÷ 24), so a flagged emergency is dispatched in about 7 minutes. False flags cost little: flag even 10% of the 2,208 others, a precision of 46%, and the priority lane carries 410 a day, 70 an hour at peak, far under 240. Missed emergencies cost the penalty. Recall is the gate; precision is a soft line.

The AI dispatcher comes second. To clear the peak it must cut handling to 3.6 minutes (24 × 60 ÷ 400), the 4-minute lookup to 1.6, a 60% cut nobody can promise before an eval exists.

Definition of done: emergency recall at least 98% on a gold set stratified to 100 emergencies and 300 others (at most 2 missed), labelled by two senior dispatchers and adjudicated; precision, reweighted to the live 8% mix, at least 70%; flag p99 under 60 seconds; then two weeks live with 98% of emergencies dispatched inside 30 minutes, up from 59%. "Every" becomes 98%, said in the room: 2% of 192 a day is about 4. Cost is not the constraint: 1,000 tokens a call, 2.4r a day.

What reverses this: timestamps showing emergencies dispatched on time and contractors arriving late.

Rubric

computes dispatcher capacity and the morning queue before proposing any build
locates the misses in queue position, with an emergency's wait at the peak worked out
chooses prioritisation first and says what the AI dispatcher would have to achieve to be first instead
sets recall as the gate and justifies a loose precision line with the priority-lane arithmetic
writes a definition of done with numbers, a labelled gold set with enough emergencies to read the recall line, and a live period, and turns "every" into a measurable line
names the evidence that would reverse the diagnosis

Where answers go wrong

"Build an agent that reads the request, finds the contract and raises the work order." Loses because it attacks 6 minutes of handling while emergencies wait 80 minutes in the queue, and it only clears the peak if it cuts the lookup by 60%, which nobody can promise on day one with no eval.

"Precision first, so dispatchers trust the flag." Loses because the lane arithmetic shows a false flag costs a few seconds of someone else's wait, while a missed emergency is a penalty credit; the interviewer wants the asymmetry stated with numbers.

"Agree to 'every emergency' and promise 100%." Loses because at 192 emergencies a day, 100% recall is not a gate any model or person can be measured against, and the COO hears about the first miss from a client before hearing it from you.