Trust, safety and regulation as product surface
Ship disclosure, consent, retention, a report button and an audit log as defaults with numbers, classify features under the EU AI Act, and name who signs.
On this page
Trust is built from five things a user can see or an auditor can check: disclosure, consent, retention, a report button and an audit log. Each is a default with a number and a named person who signs it off. Regulation arrives as constraints on those same five, so the PM classifies a feature against the text of the law before the screen is designed, not after launch.
The mistake: a principles page
The first version of "trust" for the drafting feature was a responsible-AI page in the footer and a line in the composer: "AI can make mistakes." A large prospect's security review then asked five questions. Who at our company can turn this on? Is our ticket text used to train or test anything? How long do you keep it? How does an agent tell you a draft was wrong, and what happens next? Can you show who sent a given reply and whether a person edited it?
The page answered none of them, and the deal stalled until each became a product decision, which it had been all along.
Five surfaces, each a default with a number
Disclosure. Every draft carries a "Draft" label and the help-centre article it used, so the agent knows what they are reviewing. Whether the end customer is told that a reply was drafted with AI is the tenant's setting, because the tenant is the one talking to its customers. A customer-facing assistant is a different case, below.
Consent. The feature is off until a tenant admin turns it on. Separately, and off by default, the tenant chooses whether its tickets may become eval cases. The golden set grows from field failures, and a field failure copied into the vendor's eval set is the tenant's customer's words; without the opt-in, the case is rewritten without the customer's text or dropped.
Retention. Ticket text in prompts and drafts is kept 30 days. At 200,000 drafts a day that is 6 million drafts of customer text held on any given day; 90 days would be 18 million. Thirty covers report triage, whose slowest loop is weekly, with room. The GDPR's storage-limitation principle in Regulation (EU) 2016/679 (opens in a new tab) asks for no longer than the purpose needs; the security review asks for the number. The prompt also reaches the model vendor, whose retention is a contract term the PM has read (reading an architecture).
Report button. One click on every draft, three kinds: wrong fact, wrong tone, harmful or private data. Assume agents report one in ten of the drafts they reject: 200,000 × 8% × 0.1 = 1,600 reports a day, more than anyone can read, so where each one goes is a design problem, worked in the report button.
Audit log. For every draft: who asked for it, the model and prompt version, the article ids retrieved, the edit distance before send, who sent it and when. Metadata and hashes, no ticket text, so it can be kept a year while the text is deleted at 30 days. It answers "did a person send this?" in a dispute, and it is what an execution investigation segments by when a model update moves the pass rate.
The EU AI Act as a classification exercise
Regulation (EU) 2024/1689, the Artificial Intelligence Act (opens in a new tab), sorts AI systems by what they are used for, not by how they are built. The first pass is three questions, asked in order about each feature on the roadmap:
- Is it a practice Article 5 prohibits? The list includes inferring the emotions of people at work, except for medical or safety reasons.
- Is it high-risk? Annex III lists the uses, and employment is one area: systems that allocate tasks by individual behaviour or monitor and evaluate workers' performance. Article 6 allows narrow exceptions, but a system that profiles people is always high-risk.
- Does it interact directly with people? Then Article 50 requires telling them they are dealing with an AI system, unless that is obvious.
Four features of the support tool, classified:
- Agent mood from call audio, to flag frustrated agents: prohibited, under Article 5.
- An agent quality score built from edit rates and reply times and used in performance reviews: high-risk under Annex III. Before it could ship: a risk-management system, logging, human oversight, a conformity assessment and registration, and the tenant must inform workers' representatives and the affected workers before switching it on. That is months of work, for the dashboard the support lead asked for in jobs to be done.
- A help-centre assistant answering customers before they file a ticket: Article 50 applies, so the customer is told they are talking to an AI.
- The drafted reply, read and sent by an agent: none of the three on a first reading. One open question goes to counsel: Article 50 also requires providers of systems that generate text to mark it, machine-readably, as AI-generated, with an exception for assistive editing; whether a draft an agent edits and sends falls inside it is counsel's call, not the PM's guess.
The support-tool company is the provider; each tenant is a deployer, with duties of its own. The PM drafts the classification with the articles cited, and counsel signs it, before the feature enters the roadmap: it can move the effort estimate by months, which the cost-of-delay sums need.
The NIST framework as a checklist
The Artificial Intelligence Risk Management Framework (AI RMF 1.0) (opens in a new tab) is voluntary; its four functions work as a checklist because each should point at an artefact that already exists:
- Govern: a named owner for each surface, and the sign-off table above.
- Map: who is affected (the agent, the end customer, the tenant), where the data goes, and the classification.
- Measure: the golden set with the fact class blocking at 0.5%, the report rate per 1,000 drafts, the complaint guardrail.
- Manage: the launch ramp's actions, the per-tenant flag, report triage, the incident path.
A function with no artefact behind it is the gap; it is usually Govern.
In an interview
What is being tested is whether "trust and safety" becomes screens, defaults and numbers with owners, and whether you can classify a feature against the text. "Five surfaces, each with a default. The drafted reply is not on Article 5, not in Annex III and does not talk to customers directly; the marking question goes to counsel. The agent score would be Annex III, so I would cost that before it reaches the roadmap" is an answer an interviewer can score.
The mistake that loses points is "we'd add a disclaimer and follow responsible-AI principles": a page, not a surface, with no number and no owner. The second is "the AI Act doesn't apply to us", said without classifying a single feature.
Check yourself
1. An EU tenant asks for a per-agent quality score to use in performance reviews. Build it next quarter?
No, not as specified. It monitors and evaluates workers' performance, so it is high-risk under Annex III: conformity assessment, logging, human oversight and registration first, months before it ships. Offer queue-level metrics not tied to a person, reply time by ticket type, and have counsel confirm they fall outside. If the tenant insists, score the compliant version in prioritisation with its real effort.
2. An engineer asks to keep ticket text 90 days instead of 30, for debugging. Decide.
90 days is 18 million drafts of customer text held at any time instead of 6 million, for a need the audit log mostly covers: debugging uses model, prompt and article ids, not the customer's words. Keep 30. What would change it: report data showing more than about 1 in 20 reports filed after day 21, which would mean triage needs the text longer.
3. A tenant admin wants the "drafted with AI" notice to their customers switched off. Allow it?
For drafts an agent reads and sends, yes, as the tenant's setting: the customer is dealing with the agent, and the draft label and audit log stay on for the agent and for any dispute. For the help-centre assistant, no: Article 50 requires telling the customer they are talking to an AI. Whether drafts need machine-readable marking is counsel's open question either way.