What an interviewer asks an AI engineer
Recognise the six question families in an AI-engineer loop and answer each with a number, the decision it forces and the evidence that would reverse it.
On this page
An AI-engineer loop asks six kinds of question: size it, design the loop, eval it, secure it, cost it, explain a failure. Each is a request for a number phrased as a request for an opinion. We score all six the same way, and the candidates who lose do so the same way too: they name a technique and stop.
How we score an answer
Vocabulary is free. Every candidate says retrieval, guardrails, caching and LLM-as-judge in the first five minutes, so we stop writing the words down. We write down the first number, and whether it was derived or recited. A derived number has its inputs on the board — "a 300-word ticket is 400 tokens, the system prompt is 3k" — so we can change one input and watch the answer move. A recited one cannot be probed, and it scores like a term.
Then the decision, since a number that rules nothing out was not worth computing. Then the reversal: the condition under which the candidate would choose differently, which only someone who has shipped the feature can name.
The table puts the six side by side; the sections after it say where each number was taught.
1. Size it
"Walk me through one request." We are testing whether the candidate counts before choosing. The strong answer writes the four numbers: a support-reply drafter reads about 9.1k tokens across two calls, writes 360, and has a p95 near 10 s, six seconds of it the 300-token draft decoding at the course's 50 tokens/s. The decision: ten seconds is too long for a rep watching an empty reply box, so the draft streams or is written when the ticket arrives. The reversal: if reps open tickets hours later, precompute and skip streaming.
The losing answer is "it fits in the window": no count, no output reserve, no history growth, and no thought for placement in a long context.
2. Design the loop
"The agent calls tools until it is done. What could go wrong?" We want three exits, each sized: a turn cap at twice the p95 of successful runs; a token cap, because cumulative input is quadratic in turns and a loop that runs four times too long costs thirteen times as much; and a no-progress check that hashes the tool name and arguments. Then the sentence that marks production experience: which exit fired is the first field of the trace. Agent loops and stop conditions has the arithmetic, and the tool-call loop the rule that a failed tool returns an error result instead of raising.
The losing answer is "a ReAct agent with memory": a paper and a feature, with the model as the loop's only exit.
3. Eval it
"How do you know the new prompt is better?" We are testing what the candidate thinks a score is worth. On 200 items near a 90% pass rate the standard error is √(0.9 × 0.1 ÷ 200) ≈ 2.1 points, so 185 against 182 — the revision in prompt design as interface design — is noise. The strong answer asks for the paired diff, the per-slice rates with 20 or more items each, and three repeats of each prompt, then says what the set can detect: a 2-point change at two standard errors needs about 900 items. Regression gates is the arithmetic.
The losing answer is "LLM-as-judge, tracking accuracy": a grader with no set size, so no way to call a 2-point move.
4. Secure it
"The assistant summarises web pages, and one page tells it to email the contact list." We are testing where the fix goes. The strong answer draws the path from prompt injection and least-privilege tools: untrusted text arrives as a tool result, the model sometimes obeys — 5% of the time in that lesson's assumption for an unguarded prompt — and a tool turns the instruction into an effect. The fix is that last link: the summariser's allow-list holds only fetch_page, and a refund tool has no destination argument, a bound at the order total and a person above 50. An injected request for 900 on a 120 order then pays at most 120, to the card that paid, after a person read it. The reversal is sized too: an approval threshold of 100 halves an eight-hour daily review queue and doubles the exposure.
The losing answer is "we tell the model to ignore instructions in documents". We ask about the 5%.
5. Cost it
"What does it cost a month, and how would you make it cheaper?" We are testing whether a loop is costed as a loop. Cost is tokens in × P_in plus tokens out × P_out, and the history is resent every turn: a 20-turn agent with a 4k prompt adding 1.5k a turn reads 365k tokens to write 10k, so even with P_out at five times P_in, input is seven-eighths of the bill (where the money goes). The levers, in order: shorten what repeats, drop old tool results, cache the prefix — a 12k prompt at 1,000 calls a day falls from 12 × P_in a day to (0.288 × C_w + 11.712 × C_r) × P_in, 24 writes and the rest reads (prompt caching) — then route, then batch — routing only once the cheaper tier passes the eval slice by slice.
The losing answer is "use a smaller model", said before any eval run.
6. Explain a failure
"The bot is wrong 30% of the time. What do you do on Monday?" We are testing whether the candidate measures before touching the prompt. The strong answer splits the failure with 100 questions, each tagged with the chunk that answers it. In the model does not know your data the right chunk was in the top five for 80 of them, and the model got 72 of those 80 right: 90% when it was given the answer. The 20 whose chunk never arrived are 20 of the 28 wrong answers, so retrieval was the problem; four prompt rewrites, which could reach only the other 8, had moved 30% to 28%. Every failure question has such a field: the stop reason for a summary cut off mid-sentence (reading the response), the exit that fired for a runaway loop, the cache-read counter for a bill that jumped overnight.
The losing answer is "it hallucinated, so we'd improve the prompt": a symptom named as a cause.
In an interview
Across the six we test one habit: number, decision, reversal, out loud. When a question hides its family, find it: "would you fine-tune on our help centre?" is a size-it question whose first sentence is a token count. When we change an input mid-answer, recompute rather than defend the old number. Saying the trade-off out loud is the same discipline in a system-design loop.
The mistake that loses points is a mechanism before a number — "I'd use RAG with a reranker" as the first sentence. It may be right, but it was chosen before the corpus was counted, and we cannot tell knowing from guessing.
Check yourself
1. Would you fine-tune on a 2,000-article help centre that changes weekly? Name the family and answer it.
Size it, phrased as a mechanism question. 2,000 × 3k = 6M tokens, about thirty of the course's 200k windows from the four numbers, so context is out; weekly changes make a fine-tune stale, with no citation to check. Retrieval, measured by recall at 5 on 100 labelled questions. The reversal would be a narrow task with 10k labels at a million calls a day.
2. Two minutes are left, and the question is how you would know the new prompt is better. Which two numbers do you say?
The set's resolution — 200 items near 90% is about ±2.1 points — and the rule it forces: a paired diff and three repeats before calling any move under about 4 points a change. Slices and the judge's calibration are the follow-up if there is time.
3. p95 went from 9 s to 15 s after the final answer gained a reasoning paragraph. Explain the failure and pick the fix.
Six seconds at 50 tokens/s is 300 more output tokens: the paragraph, since prefill and tools did not change. Take it off the user's path and generate it on demand, or stream so the first words arrive at the old time. A faster model is not the first move.