Support triage
Specify triage for 3,000 tickets a day as one role, then find where its saving goes: the review minute, wrong replies and mis-routes, not tokens.
On this page
Support triage reads every ticket, files it in one of 12 categories, answers the 30% it may answer alone and hands the rest to a person with a draft. Modules 4 and 8 built it a part at a time; this page specifies the whole role in order: role, SOP, rung, graph, gates, connectors, budget, evals. The arithmetic ends somewhere the token bill never would: on a $90,000 task, the saving is about $5,900 a month, and almost all of it turns on two numbers people set.
The role
This page uses a different company, as module 4 does: the online retailer's support desk from approval gates. It takes 3,000 tickets a day in an 8-hour window, 90,000 in a 30-day month; refunds are 8% of tickets, 240 a day averaging $45. Staff time is the assumed $30 an hour, so a 2-minute review costs $1.
| Field | Support triage |
|---|---|
| Inputs | About 3,000 tickets a day: the ticket and its last three messages, capped at 3,000 tokens; longer goes to a person. Supported languages only. |
| Outputs | One of 12 categories on every ticket; the reply sent when the draft scores 0.90 or more outside the must-pass categories; otherwise the draft saved for an agent. Never a refund, a cancellation, an address change or a send to any address but the ticket's own. |
| Done | Every ticket carries a category and either a sent reply or a place in a person's queue within 15 minutes. |
| Manager | The support lead: reads the samples, owns the eval set, signs off every prompt and model change. |
| Escalation | Unknown input to the support queue in 15 minutes; ambiguous rule to the owner in 4 working hours; external failure pages the owner at once. |
| Budget | About $0.0042 a ticket; $450 a month for tokens with an alarm on the daily rate; 150 seconds a ticket. |
The done line and the eval behind it come from what working means; the clocks come from escalation paths and the manager, each with a next hop when it runs out.
The SOP
Trigger A new ticket in the helpdesk.
Inputs Ticket text and last three messages; customer plan; order and account.
Steps 1. Read the ticket; strip card numbers, emails, phones; names to
placeholders; cap at 3,000 tokens. D1
2. Look up the order and the account; search the help centre.
3. One model call: a category, a draft reply, a score.
4. Check in code: category is one of the 12; draft under 400 tokens;
no refund amount or promise; placeholders fillable.
5. Route. D2 D3
6. Write the category and the draft to the ticket, keyed by ticket id.
Decisions D1 over the cap, or carrying an ID document → person, no call
D2 refund → the refund flow; cancellation or legal → a person, always
D3 score ≥ 0.90 → send the reply; below → the category's agent queue
Exceptions Unknown input (unsupported language, unreadable attachment, no order
found) → support queue, 15 min. Ambiguous (two policies apply) →
owner, 4 working hours. External (helpdesk, order system or model
provider failing) → stop calling it, hold the tickets, page the owner.
Done Every 15 minutes: tickets in = sent + queued + handed to refunds.
D2 is the decision a score must never make. Refunds, cancellations and legal tickets go to a person whatever the model's confidence, by a rule in code on the category, with a keyword check before the call as a backstop. Deterministic first is the pattern: the model labels, code decides what the label is allowed to do.
The rung, and why not the one either side
Rung 3, a graph in code. The state is a declared type, with category one of exactly 12 values and the score a number, and every path has a test that runs before a change ships. In LangGraph (opens in a new tab) terms each step is a node, the route is a conditional edge on the state, and the graph is compiled before it runs.
Not rung 2. The canvas has no test that runs before a change reaches live traffic, and at 3,000 tickets a day every change reaches 3,000 tickets the same day. The agency's renamed CRM field in rung 1 misrouted about 36 leads over three days, with no error; the same untested rename here misroutes a whole category, and refunds alone are 240 a day. Assume the code build costs 40 more contractor hours than the canvas, $3,200 at an assumed $80. The sampling rate exercise prices a change that doubles the wrong-reply rate at $1,350, found three days late; one a month caught in CI instead pays back the step in about two and a half months.
Not rung 4. The path is drawable: read, look up, categorise and draft, check, route. By the per-turn arithmetic in scripts, workflows and agents, an 8-turn agent run costs about $0.059; over 90,000 tickets that is about $5,300 a month, fourteen times the $382 the graph spends, plus three failure modes at 3,000 runs a day. The tool loop belongs below the cut, where an agent presses send: the assistant in the support desk server.
The graph
The pool is sized by throughput and latency for automations: 6.25 tickets a minute at 90 seconds each is 9.4 in flight on average. Done is 15 minutes, not the end of the day, so the pool is sized for Monday's 3× peak: 28 in flight, a pool of 30. Each step has a timeout inside a 150-second budget; a ticket that runs past it goes to a person, and past 5% timeouts in an hour the breaker sends every ticket to people until the provider recovers. The checkpointer saves each ticket's state on Postgres (Persistence (opens in a new tab)), so a deploy in the middle of the morning loses no ticket.
Gates
Triage has no approval gate on its own outputs, and the reason is module 4's inequality, error rate × cost if wrong against review cost per item:
- Sent replies: 30 of 900 wrong, about 3.3%, × $15 (a reopened ticket and a $10 credit) = $0.50, under the $1 of a 2-minute read. Sample 1 in 20: 45 a day, 90 minutes, from the sampling rate exercise.
- Routing: 4% mis-routed × $1.50 = $0.06, under $0.25 for a 30-second check. Sample 1 in 84: 36 a day, 18 minutes.
- Refunds: the refund flow's own gates, every refund over $100 and every refusal, about 60 a day, as approval gates places them.
The 0.90 cut is not a feeling. Sampling review and confidence thresholds calibrated it on 200 labelled drafts: a band belongs above the cut when its wrong rate is under $1 ÷ $15 = 6.7%. If a wrong reply ever costs more than $30, 3.3% × cost passes $1, and the replies need a gate instead: the cut moves up, not the sample. The day after any prompt, model or connector change, the sample runs at the daily rate, 1 in 3, 300 replies, $255 more for that day.
Connectors
- Helpdesk: a webhook on each new ticket, verified, stored and answered in under a second, plus an incremental poll every 5 minutes for anything it missed, per webhooks versus polling. The API allows 60 requests a minute and a ticket makes two; the 3× peak is 37.5 a minute. Every write is keyed by ticket id, so a redelivered webhook writes one category and sends one reply.
- Orders and accounts: read through the desk server with
desk:readand nothing else, under the automation's own identity, as auth and identity for an MCP server sets out. - Model provider: reached only through the redactor; about 12 MB of customer text a day crosses the tenant boundary, and card numbers and ID documents never do. Field by field, that is the data map in least-privilege credentials.
The identity is triage-bot: read tickets, set the category, send replies from the support address, read orders. No internal notes, no other send-as address, no refunds. The reply step has no recipient argument, so the forwarded email in prompt injection through your own inbox can be categorised but cannot be obeyed.
Budget
Tokens first, because they are settled. Cost control priced the capped calls at $576 a month and $382 with the 1,200-token prefix cached, about $0.0042 a ticket. The token budget exercise set the line at $450, the cached figure plus 18% and the budget who owns a virtual employee holds the owner to, with the alarm on any day over $16 or a month-to-date 20% ahead of the line; the five hourly canaries add about $15 inside it.
Then the two sums, a day at a time:
Sum 1 3,000 tickets × 2 min = 100 h × $30 = $3,000 a day
People drafts below the cut 2,100 × 2 min = 70 h = $2,100
wrong replies sent 30 × $15 = $450
mis-routes fixed 120 × $1.50 = $180
samples 45 × 2 min + 36 × 30 s = 1.8 h = $54
$2,784 a day
Run tokens $382 + canaries $15 + hosting $150 (assumed) = $547 a month
Review the owner's 20 minutes a week = $43 a month
Month $90,000 − 30 × $2,784 − $547 − $43 ≈ $5,890 saved
Build 100 contractor hours × $80 (assumed) + labels $120 = $8,120
Payback $8,120 ÷ $5,890 ≈ 1.4 months
The 900 sent replies save 30 hours a day, $900, and the automation gives about three-quarters of it back: $450 in wrong replies, $180 in mis-routes, $54 in samples, $20 to run and review. Every line here is counted as if people never mis-route or send a wrong reply, which flatters the by-hand side. The 2,100 drafts below the cut save nothing yet, because reading a draft takes the same 2 minutes as answering from a blank page.
That makes the levers plain, and none of them is a token:
- The review minute. Ten seconds off each of the 2,100 reads is 5.8 hours a day, $175, about $5,250 a month: nearly the whole current saving again. The screen that shows the draft is the cheapest build on this page.
- The mis-route rate. Each point is 30 tickets a day at $1.50, $1,350 a month.
- The cut. Covered under the mistake below.
The cascade in cost control saves $187 a month, smaller than one point of category accuracy, so it waits for the eval.
Evals
The set is the one what working means builds: 200 tickets, 150 drawn at random from the month and 50 targeted at refund disputes and the three exception classes, labelled once by the support lead in about 4 hours, $120. The acceptance line:
- category correct on at least 188 of 200, against today's 192;
- zero auto-replies on the 20 must-pass refund, cancellation and legal tickets;
- every case ends in a terminal state.
A run costs about $1.30 and gates every prompt, model, tool or category change; at 96% the standard error is about 1.4 points, so a move of 3 cases is noise and a move of 10 is not. The same 200 recheck the cut's calibration and the redactor, where one leaked card number fails the run. The mechanism is evals as the unit test of LLM work.
The set gates the changes you make; drift and silent failure watches the world changing. Rate alarms with bands sized from volume: the auto-reply share at 30%, ±3 points a day and ±8 an hour; the null share of every input field; the category mix; cost per ticket. Five canaries run hourly, one per path that must never break.
In practice
To the support team: "Triage sends 900 replies a day and we read 45 of them. It saves 30 first-touch hours a day and hands three-quarters back in wrong replies and mis-routes, so the numbers to work on are the 2-minute read and the 4% mis-route rate, not the token bill." To a vendor quoting triage: ask for the confusion table at their proposed cut on your 200 labelled tickets, with the wrong replies priced.
The mistake that costs money is lowering the cut to send more replies without pricing the wrong ones. At 0.80 the desk sends 600 more replies a day and about 90 more wrong ones; module 4's table puts that at $750 a day more, which turns a $196-a-day saving into a loss of about $550 a day.
Check yourself
1. Volume triples for one sale week. Which alarm fires, and what do you change before the week starts?
The daily spend alarm: $12.70 × 3 ≈ $38 a day against a $16 line, every day, with nothing wrong, since cost per ticket stays at $0.0042. Watch cost per ticket for the week instead. The week adds about 7 × 2 × $12.70 ≈ $178, taking the month to about $560, past the $450 stop, which would cut triage off in the last week. Raise the stop for that month in advance.
2. The sampled wrong-reply rate goes from about 3.3% to 6.7% in a week with nothing merged. Decide what to do and what waiting costs.
A doubling costs an extra 30 × $15 = $450 a day, so a week of waiting is about $3,150. Run the scheduled eval now, about $1.30, and read at the daily rate for a day. A drop with nothing merged points at the model changing underneath: pin a dated version where offered, or raise the cut to 0.95, 600 sent and about 15 wrong, until the prompt is fixed.
3. The small model would categorise for $187 a month less. Ship it before the eval?
No. One point of category accuracy is 30 mis-routes a day at $1.50, $1,350 a month, seven times the saving. Run the 200 cases through the small model and route to it only the categories where it holds.