Throughput and latency for automations

A token budget says what an automation costs; it does not say whether the automation keeps up. Support triage at the online retailer, not the agency, has 3,000 tickets to clear inside an eight-hour working day, each takes about 90 seconds end to end, and the question is how many must be in flight at once. That number sets the worker pool, and the pool decides which rate limit the automation hits first. Little's law answers it in one line; the law is derived there and only applied here.

The pool from one multiplication

arrival rate     3,000 tickets ÷ 480 minutes = 6.25 a minute
time in system   90 seconds                  = 1.5 minutes
in flight        6.25 × 1.5                  = 9.4 tickets

So a pool of 10 workers, if arrivals were flat. The 90 seconds is the whole path, assumed here from the run traces, not the model call alone:

Most of it is waiting on someone else's API. That is why a worker is cheap and why the workers themselves are rarely the binding constraint.

The mistake runs the other way: sizing from the model call alone. Count only the model's 42 seconds and the arithmetic says 6.25 × 0.7 ≈ 4.4 in flight, so a pool of 5. Five workers that actually hold each ticket for 90 seconds finish 5 ÷ 1.5 ≈ 3.3 tickets a minute against 6.25 arriving. The backlog grows by about 3 a minute and stands near 1,400 by the end of the day.

Peaks, not averages

6.25 a minute is the day's average. Arrivals are not flat; a peak hour is typically two to five times the average. Suppose the first hour of Monday carries three times: 18.75 tickets a minute, which needs 28 in flight. A pool of 10 finishes 10 ÷ 1.5 ≈ 6.7 a minute, so the queue grows by about 12 a minute and stands near 725 by 10:00. The other seven hours carry the remaining 1,875 tickets, about 4.5 a minute, so the pool works the backlog off at 2.2 a minute and clears it around 15:30.

Whether that is acceptable is a definition-of-done question, not a sizing one. If done means "triaged by the end of the day", a pool of 10 meets it. If done means the desk's promise from escalation paths and the manager, a first reply within the hour, the ticket that arrives at 10:00 waits behind 725 others, almost two hours, and the pool has to be sized for the peak: 28 in flight, a pool of 30.

Expand

Tickets arrive at 6.25 a minute on average into a queue; a pool of 30 workers, sized for the Monday peak of 28 in flight, holds each for about 90 seconds, so 9.4 are in flight on average; each ticket makes two helpdesk API calls under a 60-a-minute limit and one model call under a 60-second timeout; a ticket that runs out of its 150-second budget goes to a person100%−+FitSVG (opens in a new tab)CloseEsc

Pinch to zoom · drag to panScroll to zoom · drag or arrow keys to pan · 0 fits · Esc closes

What the pool runs into

Thirty workers cost little; what they call is the constraint. At the peak of 18.75 tickets a minute, with two helpdesk calls and one model call per ticket:

helpdesk API    18.75 × 2         = 37.5 requests a minute
model calls     18.75 × 1         = 18.75 requests a minute
model tokens    18.75 × 2,450     ≈ 46,000 tokens a minute

Assume the helpdesk allows 60 requests a minute per API key: 37.5 fits, with room for the connector's own retries. That limit, and your account's requests-per-minute and tokens-per-minute limits at the model provider, are the real ceilings. Size the pool below the lowest of them, and give every worker the backoff on a 429 described in rate limits, retries and backoff. A pool larger than the limit allows only turns waiting in your queue into retries against theirs.

Batch or real time

Not every automation has someone waiting on it. There are two shapes:

Real time. Each item is processed on arrival, the pool is sized by Little's law at the peak, and latency is minutes. This is right when a person waits on the result — the agency's lead handler and its five-minute clock, a support ticket's first reply.

Batch. Items collect and run on a schedule. Bookkeeping prep's 600 transactions a month are about 20 a day; run nightly, one worker clears them in 30 minutes (20 × 90 s), and the pool, the peak and most of the rate-limit question disappear.

Batch whenever the definition of done tolerates the delay. A batch can be retried as a whole, tested on yesterday's input before it runs on today's, and scheduled for the hours when the rate limits are idle. Some providers also price an asynchronous batch interface below their real-time one; read your provider's terms before counting on it.

Timeouts are budgets

Little's law cuts both ways: if time in system grows, so does the number in flight. Many HTTP clients wait minutes for a response by default. Suppose the model provider has a slow hour and calls hang for 10 minutes before failing. Then 6.25 × 10 ≈ 63 tickets want to be in flight, the 10 workers are all stuck within two minutes, and every ticket behind them waits for calls that will fail anyway. The same effect, a slow dependency exhausting a pool, is the subject of the Little's law lesson.

The fix is to treat the 90 seconds as a budget and give each step a share of it, with a timeout at that share:

A ticket now holds a worker for at most 150 seconds, not 10 minutes. A timed-out step returns its error as data and the ticket goes to a person; a run of them is the external-failure class, which pages the owner at once. If more than 5% of tickets time out in an hour, stop taking new tickets into the automation and route them all to people until the provider recovers — the circuit breaker pattern, applied to a model call. Setting a model call's timeout from its output cap is worked in retries, timeouts and rate limits.

In practice

Ask any vendor running an automation for you three numbers: the concurrency they will run at, the rate limits they assume on each system they call, and the timeout on each step. Tell the team the response promise in minutes, so the pool can be sized for the peak that promise has to survive, not the average.

The mistake that costs money is sizing the pool from the model call and the daily average. The backlog it builds is paid for twice: once in customers who wait, and again in the people who clear the queue by hand.

Check yourself

1. A new lookup step raises the time per ticket from 90 to 150 seconds. What pool does triage need now, at the average and at a 3× peak?

Average: 6.25 × 2.5 = 15.6 in flight, a pool of 16. Peak: 18.75 × 2.5 ≈ 47. Check the helpdesk limit before adding workers: if the new step is another helpdesk call, the peak makes 18.75 × 3 = 56 requests a minute against a 60 limit, with almost no room for retries.

2. Triage could instead run overnight, 3,000 tickets between 01:00 and 05:00. Does it fit the 60-a-minute helpdesk limit, and how big is the pool?

3,000 ÷ 240 minutes = 12.5 tickets a minute, 25 helpdesk requests a minute: it fits. In flight, 12.5 × 1.5 ≈ 19, a pool of 20. Whether triage may wait until morning is the real decision — a customer who wrote at 09:00 hears nothing until the next day, so overnight suits the backlog, not the daytime stream.

3. The model-call step's p99 is 40 seconds. Where do you set its timeout, and what happens when it fires?

A little above the p99, say 60 seconds, the share the budget gives it. When it fires, the ticket goes to a person as an external failure and the timeout counts towards the hourly rate; past 5% in an hour, the breaker routes everything to people. A timeout under the p99 would fail a percent of healthy calls on purpose.