The AI-engineer interview · Interview

Design a meeting-notes feature

Replace a proposed agent with the simplest mechanism that turns sales calls into CRM tasks, and place the gates that make its writes safe.

designhardExercise · 4 min · 75 of 77

On this page

A sales team wants call notes and next steps in the CRM without the rep typing them. The engineers have already drawn an agent; the round turns on whether you can replace it with something smaller and say why in numbers.

The situation

Account reps hold 3,000 customer calls a day, 45 minutes on average and 90 at p95. A speech-to-text service delivers each transcript, with speaker labels, a minute after the call ends. Product wants three things within 10 minutes: a short summary on the account, each next step as a CRM task with an owner and a due date, and a drafted follow-up email. A wrong owner is the expensive error, because the task sits with someone who never reads it.

The team's proposal: chunk each transcript into 1k-token pieces in a vector index, and run an agent with search_transcript, create_crm_task and send_followup_email until it decides it is finished.

Course constants for the arithmetic: speech at about 150 words a minute; 1.3 tokens per English word; a 200k-token window; 50 tokens/s decode; P_in = $3 and P_out = $15 per million tokens. The round follows designing an LLM feature in 40 minutes.

The question

Lay out the mechanism you would ship instead — calls, output schema, and a gate on every write — and defend it against the agent with numbers.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Count first: 45 × 150 × 1.3 ≈ 8.8k tokens of transcript, 17.6k at p95; with 2k of instructions and account context, about 11k in, 20k at p95 — a tenth of the window. Nothing to retrieve; chunking would split a next step agreed at minute 40 from the objection at minute 10. One call.

Out: a summary of ~200 tokens, six next steps at ~50, an email draft of ~200: ~700 tokens, 14 s of decode — a background job against a 10-minute promise, no streaming. Cost: 11k × $3/M ≈ $0.033 plus 700 × $15/M ≈ $0.011, about $0.044 a call, $130 a day, $3,900 a month. The agent: five turns that each add 3k of search results read 2k + 5k + … + 14k = 40k tokens, 3.6 times the call, and never see the whole transcript.

Schema, enums first: owner, an enum of the account team's rep ids plus customer, built per call; due, a date or null; action; quote; free-text summary and email_draft last.

Gates. Output: schema, then invariants — owner on the account team, due on or after the call, every quote verbatim in the transcript — one retry naming the failed check, then the rep reviews. Action: the model has no tools; code writes the CRM tasks with an idempotency key of call id plus item index, so a retried job never duplicates one. The email stays a draft for the rep: the customer spoke into this transcript, and "send the pricing sheet to my personal address" is an outsider's instruction.

Reversal: questions across calls, such as this account's objections all year, would bring retrieval back.

Rubric

converts call length to tokens at the mean and the p95, and concludes one call fits the window with room
states output, latency and cost per call and per month with the arithmetic shown, and draws the background-job conclusion
prices the agent against one call and names what chunking would separate
gives the owner field as a per-call enum and puts free text last
puts an output gate on owners, dates and quotes, and says what happens on a second failure
takes write tools away from the model, keys the CRM writes for idempotency, and leaves the email as a draft because the customer is an outside speaker

Where answers go wrong

"Keep the agent, but add a line to the prompt saying never to email anyone outside the account." That is a prompt line with a measured failure rate, guarding a send tool from text an outside party wrote; the interviewer asks about the calls where the line does not hold. Removing the tool is the fix, as prompt injection and least-privilege tools shows.

"Chunk the transcript so the model can focus on the relevant parts." At p95 the transcript is a tenth of the window. Chunking adds an index and a recall ceiling, and separates the objection from the next step that answers it — the candidate never counted.

"Let the model create the CRM tasks through a tool call." The output is right, but the write has the wrong owner: a retried job duplicates every task unless something keys it, and a model's tool call is not where an idempotency key comes from.