The four shapes of an AI deployment
Choose between batch classification, a retrieval-backed assistant, an agent with tools and a human-in-the-loop queue from latency, volume and how reversible the action is.
On this page
Nearly every deployment an FDE ships is one of four shapes, and the shape is decided by three questions about the workflow, not by what the model can do. Is a person waiting for the answer? Does the output act on something, or does a person read it? If it acts, can the action be undone? The answers fix the latency budget, the queue, where the human sits and what the eval has to measure, before any prompt is written. The mistake that costs weeks is picking the shape from the demo that impressed the sponsor and then discovering the workflow needed a different one.
The three questions
Is a person waiting? If nobody is looking at a screen when the model runs, latency is free. A claim logged at 09:14 that is triaged at 09:20 costs the handler nothing, because the handler picks it up at 09:40. If someone is typing and watching, the budget is the seconds a person tolerates before they switch tabs, and p99 is the number, not the mean, for the reason tail latency gives: at 1,200 claims a day, the p99 is twelve handlers' worst moment every day.
Does the output act? A summary a person reads is cheap to be wrong about: the reader notices, shrugs, and does the work the old way. A routing decision, a field written to the claims system, an email sent to a policyholder: these change state somewhere else, and the cost of a wrong one is whatever it takes to find it and reverse it.
Can the action be undone, and by when? Re-routing a claim to the right desk costs a day. A payment released to the wrong account costs a recovery process. A letter telling a policyholder their claim is declined cannot be unsent. The less reversible the action, the closer a person has to sit to it.
Shape 1: batch classification
Nobody waits, nothing acts directly, and the volume is known. Documents arrive, the model labels, extracts or scores each one, and the results land in a table someone reads later. Invoices coded to a ledger account overnight, support tickets tagged every fifteen minutes, 1,200 claims given a provisional route before a handler picks them up.
The architecture is a job, not a service: a scheduled run, a loop over new rows, a results table and a retry list. The whole of batch vs stream applies: the staleness the workflow tolerates decides the window. Handlers who start at 08:00 and pick claims up until 18:00 tolerate a 15-minute window; a nightly run would leave the 09:00 claims untouched for the whole morning. Volume is the easy part. 1,200 claims a day is 0.033 per second averaged over the working day, and the three-hour peak worked in sizing the deployment is still under one call in flight.
What it is not good for: anything where the person needs the answer while they are looking at the case.
Shape 2: retrieval-backed assistant
A person is waiting, and the output is read, not acted on. The handler opens a claim, the system retrieves the two or three governing clauses from the 340-page policy set and shows them beside the form. The handler still decides. The claims pilot's clause panel on its own would be this shape; it becomes shape 4 because the same record also proposes the route. The model's job is to shorten the 7 minutes of searching that the as-is map in map the workflow before the model found, and nothing else.
This is a service with a latency budget. A handler who has just opened a claim will wait up to about 20 s, the tolerance the definition of done observed; past that they open the PDF themselves and the pilot measures nothing. The parts around the model are ordinary services: a cache keyed on policy version, a timeout, and a fallback to the old search box when the model is slow, in the shape circuit breakers describes. The eval is retrieval hit rate against the clause the handler chose, which their own click records for free.
What it is not good for: a workflow where the person would act on the answer without checking it, because then it is shape 4 wearing shape 2's clothes.
Shape 3: agent with tools
The output acts, a person may or may not be waiting, and the actions can be undone. The model is given tools (look up the policy, fetch the claim history, draft a reply, create a task) and loops until it has done the job or given up. The canonical example is not customer-facing at all: a reconciliation agent that opens a mismatched invoice, pulls the purchase order, and files a correction ticket, which a person can close without effect.
Every tool call is a side effect that may be retried, so every tool the agent can call needs an idempotency key, for the reason idempotency derives from at-least-once delivery: the loop will call create_ticket twice when the first response is lost, and the second call must not create a second ticket. The agent's budget is a step count and a token count per task, both capped, and the eval is task completion on a set of recorded cases rather than a per-answer score.
What it is not good for: any action without an undo. An agent that can release a payment is a shape-4 problem that someone forgot to put a person into.
Shape 4: human-in-the-loop queue
The output acts and the action cannot be cheaply undone, so a person signs each one. The model produces a draft decision with its reasons; the draft sits in a queue; a person approves, edits or rejects it; the approved decision is what acts. The claims-triage pilot is this shape: the triage record proposes a route and the governing clause, the handler confirms it, and the confirmation is what moves the claim. A wrong desk costs a day; a fraud-review claim fast-tracked costs a payment, which is why the person signs.
The queue is the design. It is an ordinary work queue with the properties message queues lists, plus two things specific to the shape. First, the person's decision is recorded beside the model's draft, so every working day produces labelled cases: at the 400 claims a day the first two assisted teams handle, two weeks is 4,000 handler decisions. They are anchored by the draft, so they measure trust as well as correctness, and the 200-case gold set stays blind-labelled from history, as building the eval set does it. Second, the queue has to survive the model being wrong, slow or down: a claim whose draft never arrives still reaches a handler, with an empty draft, inside the same 15 minutes. That fallback is the difference between a pilot that pauses and one that stops the claims floor.
The hours a person spends signing are the pilot's main cost and its main argument. A handler who confirms a draft in 4 minutes against 11 unassisted has saved 7 minutes a claim; one who takes 9 minutes because the draft is wrong one time in four has saved 2 minutes, 18%, short of the 3.3 minutes the 30% target needs, and the rollout gates in crawl, walk, run will say so.
Shapes are not forever. After four weeks of handlers approving 97% of drafts untouched in one segment, that segment can move to batch with a one-in-twenty sample. The transition to watch for is 2 to 4 by accident: an assistant whose suggestion is accepted in under five seconds is being signed, not read, and there is no queue, no signature record and no fallback behind it.
In an interview
The case round is testing whether you reach for the questions before the shape. The interviewer describes a workflow and the strong answer asks three things, out loud and in order: what the output changes, who is waiting, and what it costs to reverse. Then it names the shape and the one thing the shape makes expensive: "nobody is waiting and the route is a label the desk reads and corrects the same day, so this is batch classification with a 15-minute window, and the cost is a results table the handlers actually look at." Naming what you are not building scores as highly as what you are: "not an agent, because a wrongly declined claim has no undo."
The mistake that loses points is choosing the shape from the model's capabilities rather than the workflow's reversibility. "It can call tools, so it should" produces an agent that releases payments; "it is a chat model, so a chatbot" produces an assistant that is acted on without a queue. The second mistake is sizing for shape 2 latency when nobody is waiting, which spends the pilot's weeks on a p99 that has no one to feel it.
Check yourself
1. A bank's dispute team gets 300 chargeback requests a day. The model drafts the customer letter, and letters go out at 17:00 in one batch. A wrongly sent decline letter triggers a regulatory complaint. Which shape?
Nobody is waiting (the batch goes at 17:00), the output acts (a letter is sent), and the action cannot be undone. Shape 4: a human-in-the-loop queue where a dispute officer approves each draft before the 17:00 send. Not batch classification, even though the schedule looks like a batch job: the schedule is about when letters leave, not about whether a person signs them. At 300 a day and, say, 2 minutes a review, that is 10 officer-hours a day, which is the number the sponsor needs to hear before agreeing.
2. The sponsor wants "the assistant to just file the routing when it is confident". When is that a shape change you would accept?
When the queue's own data justifies it for a segment, not a confidence score. Four weeks of handlers approving the draft unchanged in 97% of motor claims under the small-claims limit is evidence; a model's self-reported confidence is not, because it is not calibrated against their cases. Accept it as shape 1 for that segment only, with a 1-in-20 sample still going through a handler, and keep the other segments in the queue. If the correction rate in the 1-in-20 sample rises above 1 in 25, against the 1 in 33 it entered at, the segment goes back to the queue.
3. Handlers accept the assistant's suggested clause in a median of 3 seconds. What has happened, and what do you change?
They are signing, not reading: 3 seconds is not long enough to check a clause against a form. The assistant has become a shape-4 system without a queue, a signature record or a fallback. Change the UI so accepting the clause is an explicit action that is logged against the handler, add the empty-draft fallback so a slow model does not block the claim, and start measuring agreement between the accepted clause and a reviewer's choice, because the clicks you were using as an eval signal now measure trust rather than correctness.