Where the model sits
Place the model as a hosted API, a listing in the customer's cloud tenant or servers on their premises, and say what each does to latency, cost, residency and the on-call rota.
On this page
The model can be called in three places: a hosted API run by the model vendor, the same model offered as a listing inside the customer's own cloud tenant, or servers the customer runs themselves. The pilot service, the data and the queue stay inside the customer's tenant in all three; what moves is the one call that carries the prompt. That call decides four things the customer will ask about in the first week: how long it takes, what it costs, where the data goes, and who gets paged when it fails. The shape of the deployment from the four shapes rarely changes with the placement; the contract, the questionnaire and the rota do.
Latency: the round trip is not the number
The intuition that on-premises is faster is right about the network and wrong about what dominates. A round trip inside a datacentre is about 0.5 ms; a round trip from India to the US East coast is about 200 ms. Against that, a triage call generates 160 output tokens, and at an assumed 50 tokens per second of output that is roughly 3 s of decoding on top of reading 4,300 tokens of input. The call takes seconds wherever the model sits; the region adds a fifth of one second.
Where the placement does change latency is through capacity. A hosted API has a rate limit in tokens per minute set for your account, on capacity the vendor shares across its customers, and the pilot's peak worked in sizing the deployment is about 1M input tokens an hour, under 20k a minute, which sits inside any ordinary limit. A model running on the customer's own two GPUs has whatever capacity two GPUs have, and the first week that marketing runs a 40,000-document backfill on the same servers, the triage queue's mean goes from 8 s to 80 s and its p99 far past the 20 s line. That is the latency question to ask: not "how far is the model" but "who else is using it".
Cost: r is the same, the multiplier is not
The hosted API charges r per million input tokens and 4r per million output, and the triage pilot's 5.2M input tokens and 0.19M output tokens a day cost about 6r a day, which the customer multiplies by their own r and nobody argues about.
The listing in the customer's cloud tenant usually prices the same tokens at the same r, billed through the cloud account the customer already pays, which is often the point: it is a line on an invoice they already approve, so procurement is a week shorter. What it adds is the cloud's own charges around the call, and what it removes is a vendor to onboard.
On-premises replaces a per-token rate with hardware that runs whether or not claims arrive. Work out the utilisation before anyone quotes a server: 1,200 claims a day at a mean of 8 s of model time each is 9,600 s, or 2.7 hours of serving a day. A server bought for the three-hour peak is busy about 11% of the time, and that is before the model is swapped for an open-weights one whose quality on the customer's 200-case eval is not yet known. The number to put in front of a CIO is not the hardware price, which they can get themselves; it is "2.7 serving-hours a day on hardware that needs to be sized for 240 claims an hour at 09:00".
Residency: what leaves, and what the questionnaire asks
With a hosted API, the prompt leaves the tenant. For the triage pilot the prompt contains a claim form, an adjuster's note and a policy section, which is personal data under most regimes, and the security questionnaire from the questionnaire will ask where it is processed, whether it is retained, and for how long. The engineer's answers are the vendor's published data-handling terms: the region the request is processed in, the retention period for inputs and outputs (by default a window, commonly up to 30 days, for abuse monitoring; zero only under a separately approved zero-retention agreement), and whether inputs are used for training (no, under a business contract, and say which clause).
The listing in their tenant keeps the prompt in their region and under their cloud agreement, which answers the residency question with a document the customer already signed. It does not answer the retention question by itself; the listing has its own logging, and the pilot's own logs are the more likely leak, since they are the ones you wrote at 23:00 with the full prompt in them.
On-premises answers residency completely and moves every other question onto the customer: patching, model updates, the GPU driver that breaks on a Tuesday. The CIO who wants on-premises for residency is usually right about the requirement and wrong about the cheapest way to meet it, and the honest answer names the in-tenant listing as the middle option before agreeing to anything.
Who is on call
This is the question the engineer forgets and the customer's operations lead does not. For the hosted API, the vendor's status page is the on-call, and the pilot's runbook says what the queue does when the API returns errors for ten minutes: drafts stop, handlers keep working on empty drafts, nobody is paged at 02:00 because nothing the customer runs has failed. For the in-tenant listing, the cloud's support contract is the escalation, and the customer's cloud team is already on its rota.
For on-premises, someone on the customer's staff carries the pager for a model server, and in most deployments that person has not been named when the decision is made. The handover doc in runbooks and the handover doc has a line for it; if the line is empty in week 1, the deployment will not be on-premises in week 8, whatever was agreed.
The decision in one table
| Hosted API | Listing in their tenant | On their premises | |
|---|---|---|---|
| Round trip | +200 ms cross-region | +0.5 ms, same region | +0.5 ms |
| Capacity | vendor's rate limit | their quota | their GPUs, shared |
| Cost | r, 4r per million | usually the same r, on their cloud bill | hardware, ~11% busy at 1,200 claims/day |
| Prompt leaves the tenant | yes | no | no |
| Retention | vendor's terms, cite them | cloud's terms plus the listing's logs | their logs |
| On call | vendor status page | cloud support, their cloud team | a named person on their staff |
| Model swap | vendor's schedule | listing's schedule | theirs, re-run the eval |
Default to the hosted API for the three-day prototype whatever the eventual placement, because the prototype's job is to show the workflow working on their data by Thursday, and the placement decision can be made with the sizing numbers the prototype produces.
In an interview
The customer-scenario round often opens with "the CIO wants it on-premises", and it is testing whether you can separate the requirement from the solution without arguing with the CIO. The strong answer names the requirement (the prompt must not leave their jurisdiction, or their retention policy must apply), names the three placements, and shows the middle one meets the requirement with a document they have already signed. Then it does the utilisation arithmetic out loud: 2.7 serving-hours a day on a server sized for the 09:00 peak.
The mistake that loses points is treating placement as a latency decision. The 200 ms round trip is a fortieth of an 8 s call; the interviewer wants to hear residency, retention, capacity under someone else's load, and who is paged. The second mistake is agreeing to on-premises in the room: an engineer who says "yes" before the eval has run on the open-weights model has promised a quality number nobody has measured.
Check yourself
1. The customer's cloud offers the same model as a listing in their tenant at the same r. The hosted API is already set up for the prototype. The sponsor asks which the pilot should use. Decide.
The listing. Latency is the same to within 200 ms on a multi-second call, cost is the same
r, and the listing answers the residency and procurement questions with agreements the customer already holds. The cost is a week to get the listing enabled and the quota raised to 20k tokens a minute, which fits inside the two weeks the export pipeline takes anyway. The evidence that reverses it: the listing lags the hosted API on a model feature the prompt depends on, which the 200-case eval would show as an agreement gap.
2. The CIO insists on their own servers. Estimate what the servers do all day at 1,200 claims and an 8 s mean, and say what you tell her.
1,200 × 8 s ≈ 9,600 s ≈ 2.7 hours of serving a day, with 60% of it between 09:00 and 12:00, so capacity is set by a 240-claims-an-hour peak and idle about 89% of the time. Tell her the requirement she has (the prompt stays in the jurisdiction) is met by the in-tenant listing, that on-premises adds a named on-call and a model whose quality on her 200 cases is unmeasured, and that if she still wants it, the pilot runs on the hosted API or the listing first and the on-premises move is a separate phase with its own eval gate.
3. The hosted API returns errors for 20 minutes at 09:30, the peak. What should the queue have done, and how many claims did it touch?
20 minutes at 240 claims an hour is about 80 claims. The queue should have delivered those 80 to handlers with an empty draft inside the usual 15-minute window, logged the 80 claim ids for a re-run, and paged the IT on-call, as the drafts-stalled entry in runbooks and the handover doc does in working hours: to confirm the fallback is serving and to run the re-run job once the API recovers. Nothing the customer runs had failed. If instead the 80 claims waited for the model, the pilot stopped the claims floor for 20 minutes at its busiest hour, which is the story the blocker will tell for the rest of the engagement.