Reading an architecture
Follow one request through gateway, service, cache, retrieval and model call, and ask the four questions that locate latency, failure, cost and customer data.
On this page
An architecture diagram for an AI feature has five boxes, and a PM who can follow one request through them runs the meeting instead of attending it. The running example is the drafting feature: 12,000 agents across all tenants, 200,000 drafts a day, each a 1,500-token prompt and a 300-token reply. Four questions turn the diagram into product decisions: where is the latency, what fails, what does one call cost, where is the customer's data.
Five boxes, one request
The agent clicks "draft". The request crosses an API gateway, which checks the session and the tenant's rate limit in about a millisecond. The draft service looks in a cache for a draft already made for this ticket (a hit answers in 0.5 ms), queries the retrieval store for the five help-centre chunks closest to the ticket (about 80 ms), assembles the prompt, and streams it to the model vendor's API. Tokens stream back over about two seconds, forwarded to the console as they arrive.
The sizing is back-of-envelope estimation: 200,000 requests a day ÷ 86,400 s ≈ 2.3 a second on average; support traffic peaks in the working day, so plan for 2–5× that, call it 10 a second. Every box except the model handles that without noticing.
Question 1: where is the latency
Add up the path. Gateway, 1 ms. Cache lookup, 0.5 ms. Retrieval, 80 ms. Model, 2,100 ms at p50 to the last token, 6,000 ms at p95. The model is 96% of the median request and 99% of the slow one. Nothing else on the diagram is worth an engineer's week until the model's share is handled, and the handle is not a faster model, which is the vendor's problem, but a change to what the agent waits for.
The first token arrives at about 400 ms and is on the agent's screen at about 500 ms; the agent reads while the model is still writing. The interface is designed around that first number (the non-deterministic UI) and capacity around the second: by Little's law, 10 requests a second × 2.1 s = 21 streams open at the median, and 10 × 6 s = 60 at the p95. Sixty open connections to one vendor is the figure to put beside the vendor's concurrency limit, and tail latency is why the p95 is the one to plan on.
Question 2: what fails
Each box fails differently, and each failure needs a product decision.
- The gateway rejects a request over the tenant's rate limit. Decision: does the agent see "try again in a minute", or does the limit sit above any real peak?
- The retrieval store times out. Decision: draft without chunks and say so, or no draft. An ungrounded draft is the one most likely to invent a policy.
- The model returns an error, a rate-limit response, or nothing for 30 seconds. Decision: retry once, show the empty composer with a one-line reason, or fall back to a second model class. A circuit breaker stops 60 open streams becoming 600 while the vendor is down.
- The model answers, but refuses, or returns text in the wrong shape. Decision: what the screen shows. Designing for "the model might be wrong" owns this one.
One chain to complete every time a retry appears: a retried model call is at-least-once delivery, at-least-once means duplicates, and duplicates mean the call needs an idempotency key. Without one, a retry after a slow success bills the feature twice and can post two drafts into the composer.
Question 3: what does one call cost
Only one box sends an invoice per call. The cache, the retrieval store and the gateway cost what the servers cost: fixed, and small at 10 requests a second. The model call is metered by token: 1,500 in and 300 out, which at an assumed $3 per million input tokens and $15 per million output is $0.0045 + $0.0045 = $0.009 a draft, $1,800 a day. Tokens as a cost model does the month and the levers. The box with the latency is also the box with the bill, so every lever on one is a lever on the other. The retrieval store's embedding calls are a few hundred tokens at a far lower price class, and re-embedding the help centre is a weekly job; both round to zero next to $1,800 a day.
Question 4: where is the customer's data
Walk the diagram again following the ticket text rather than the request. It is in the gateway's access log (tenant id and timing, not the body, if the engineers have done it right). It is in the cache, keyed by ticket. It is the query to the retrieval store, which also holds the tenant's help articles. It leaves the company's network in the prompt to the model vendor, with the five chunks. The vendor's retention policy for prompts is a contract question, but the PM has to know the answer before the security review asks.
The decision here is isolation. One vector store per tenant, or one store with a tenant column? The first costs more to run and makes a cross-tenant leak a bug that cannot happen; the second is cheaper and makes it a bug that can. For a product sold to support organisations with contract clauses about customer data, the first. Auth and the boundary covers the same question from the deployment side; encryption covers what "encrypted at rest" does and does not protect.
The script for the meeting
The four questions are a script. "Walk me left to right. Which boxes do we own, and which do we pay per call? Where does the ticket text go, and how long does each box keep it? If the vendor is down for ten minutes at 4 pm, what does the agent see?" An engineer answers in five minutes, and the answers are the first draft of the AI-feature PRD's requirements: a p95, a cost per request, a failure-mode table, a data-handling line. The PM designed nothing; the PM read a diagram and wrote down the decisions it implied.
In an interview
What is being tested is whether you can find the product decisions in a box diagram, not whether you can draw one. Walk it left to right out loud, give each box its number ("about a millisecond, about 80 ms, about two seconds"), say which box carries the cost, then say what the agent sees when each box fails. "The model is 96% of the latency, so we stream, and we design the screen around the 500 ms first token rather than the 2.1 s last one" is the sentence of a PM who can read a system.
The mistake that loses points is treating the diagram as an engineering artefact and skipping to the user story. The second is optimising the wrong box: a candidate who proposes caching retrieval results to save 80 ms, when the model call is 2,100 ms, has not done the addition. The third is leaving the data question for the interviewer to raise: a PM who cannot say where the ticket text goes cannot answer a customer's security review.
Check yourself
1. The vendor's p95 rises from 6 s to 12 s for an afternoon. Traffic is 10 requests a second at peak. What happens to the service, and what does the agent see?
Little's law: 10 × 12 s = 120 streams open at the p95 instead of 60; a pool sized for 60 queues the rest until they time out. The median request still gets its first token at about 400 ms, so most agents notice nothing; the slow tail sees the "slow" state: a cancel control after 4 s, composer still usable. The decision to take to the engineers: cap concurrent streams per tenant, so one tenant's burst cannot queue another's drafts.
2. The engineers propose caching drafts by ticket for an hour to cut cost. Each ticket is drafted 1.1 times on average. Do you take it?
A hit saves one $0.009 call; at 1.1 drafts a ticket, one draft in 11 is a repeat, about 18,000 hits a day, $164 a day, about $4,900 a month against a $54,000 bill. Ask first what a stale hit looks like: the agent edited the ticket, asked again, and got the old draft. Key the cache on the ticket's last-modified time, take it, and measure the hit rate before counting the saving.