Skip to content
echogeco
Running it: evals, cost and security · Security

Least-privilege credentials and data leaving the tenant

Give each automation its own identity and narrow scopes, rotate credentials cheaply, and draw the data map of what reaches the model provider and for how long.

Lesson · 7 min · 58 of 75
On this page

Every automation acts with someone's credentials, and every model call sends your data to a company that is not yours. Two questions bound the damage when something goes wrong. Which identity does the automation run as, and with which scopes? And what data reaches the model provider, how long is it kept there, and what was removed before it left? Both answers fit on one page per automation, and both are usually missing.

Everything runs as the founder

The common starting point, at the agency from how to use this course: six automations — lead handling, support triage, bookkeeping prep, the weekly report, the content pipeline, the recruiting screen — each connected by the founder clicking "allow" on a consent screen. Each one now holds the founder's own grant: the whole mailbox, every file in the drive, admin on the CRM, owner on the accounting system.

Four things follow, none visible until it happens:

  • The blast radius is everything the founder can do. An injected triage run, the case in prompt injection through your own inbox, can reach the payroll spreadsheet, because the founder can.
  • The audit log says the founder did it all. Nobody can tell a 02:00 bulk export by an automation from one by a person.
  • One password change stops six automations at once. So does the founder leaving.
  • Nothing can be revoked alone. Suspect the recruiting screen and the only switch also turns off triage.

One identity per automation

Each automation gets its own identity — a service account, an app registration or a dedicated user — named for the role (triage-bot, books-bot) and owned by the role's owner. The audit log now names the automation. Revoking one leaves the other five running. How a token is issued, refreshed and scoped is covered in how systems talk: OAuth, API keys and scopes; the same rule for a server you build is auth and identity for an MCP server.

The scopes come straight from the action allow-list. If the retailer's triage may set a category and send a reply, it needs those writes, the reads behind them, and nothing else. Two of the agency's automations follow the same rule:

AutomationIdentityScopesNever
Retailer's support triagetriage-bothelpdesk: read tickets, set category, send replies from the support address; orders: readinternal notes, any other send-as address, file storage, refunds
Bookkeeping prepbooks-botbank feed: read; accounting: write draft categoriespost or approve entries, payments, payroll
Lead handlingleads-botform: read; CRM: create and update leads; mail: send from the sales inboxdelete or export contacts

Read scopes on a whole drive or a whole mailbox are the ones that leak, because everything an automation can read is something an injected instruction can ask it to repeat. The general case, authorisation as a scope per caller, is in authentication and authorization.

Where credentials live, and how often they change

Credentials live in a secrets store, read by the automation when it starts — never in a workflow's exported JSON in a repository, a spreadsheet of keys, or an environment file on a laptop.

They rotate on a schedule — every 90 days — and on events: a person with access leaves, a key appears somewhere public, a vendor reports a breach. Rotation happens only when it is cheap. With one credential per automation, rotating triage's key touches one workflow and takes ten minutes. With one admin key shared by six automations, rotating breaks all six, so nobody does it, and a two-year-old admin key sits in six places.

A credential that expires without warning is a silent failure of the kind in drift and silent failure: the runs start failing, or worse, a connector starts returning empty results. Alarm 14 days before each expiry, to the role's owner.

The data map

The second question is about data, not access. Take the online retailer's triage from approval gates, not the agency. Every call sends the ticket to the model provider: about 1,000 tokens of customer text, roughly 4,000 characters. At 3,000 tickets a day that is 12 MB of customer text a day leaving the tenant, about 360 MB a month. The data map says, field by field, what goes and in what form:

FieldCategorising needs it?Drafting needs it?Sent as
customer namenoin the greetinga placeholder, filled in after the call
email address, phonenonoremoved
order numberyesyessent
card number, even partialnonoremoved
street addressnorarelyremoved; a person handles address changes
a photo of an ID document (age-restricted orders)nononever sent: those tickets go to a person
the rest of the ticket textyesyessent, after the above are stripped

The redactor runs inside the tenant, before the call: pattern rules for card numbers, account numbers, government IDs, emails and phone numbers, and names swapped for placeholders such as CUSTOMER_NAME that the harness fills in when the draft comes back. The model never needs the name to choose a category, and the reply still greets the customer by name.

One arrow crosses the boundary. Everything the map decides is about what is on that arrow and how long it stays on the other side.
100%SVG (opens in a new tab)

Pinch to zoom · drag to pan

Data persists in three places, and the map records each:

  • At the model provider, for whatever its terms say. Read your provider's data-usage and retention terms, write the number of days into the map, and note whether your inputs may be used for training. Some providers offer shorter retention on request; it is a contract question, not a setting you assume.
  • In your own trace store, which keeps payloads for debugging. Keep them redacted and for a fixed period, 30 days here, as observability and tracing sets out.
  • In the source system, unchanged; the automation adds a category and its replies, nothing more.

Redaction has an error rate, so it is tested. Run the redactor over the 200 cases in the eval set from what working means and count what got through. A leaked card number is a must-pass failure, not a percentage.

In practice

Ask anyone who builds or runs an automation for you four questions: which identity it runs as; which scopes that identity holds, and why each one; where the credential is stored and when it last rotated; and which of your fields reach which model provider, kept for how long. The data map is also the page you hand a customer's security questionnaire.

The mistake that costs money is building every automation on the founder's own login. It is the fastest way to connect a tool and the most expensive way to discover, during an incident, that one leaked token could read everything the company owns.

Check yourself

1. Six automations share one admin API key for the CRM, and the key has just appeared in a public repository. What do you do in the next hour, and what does it cost?

Revoke the key at once: all six automations stop, because they share it. Issue six new credentials, one per automation, each with only the scopes it needs, and reconnect them one at a time, starting with the ones customers wait on. At ten to twenty minutes each that is one to two hours of outage on everything — the price of the shared key, paid at the worst moment. Then read the CRM's audit log for what the leaked key did.

2. Bookkeeping prep sends each bank transaction to the model: date, amount, counterparty name, memo line, and the account and routing numbers of both sides. Which fields does categorising need?

Amount, counterparty name and memo carry the category; the date occasionally helps with recurring items. Account and routing numbers carry none of it, so they are removed before the call. Counterparty names stay, since "office rent" and a landlord's name are the signal. Write the decision into the map, field by field.

3. The redaction check on the 200-case set finds one card number that reached the model, written in an unusual grouping with spaces. Card numbers appear in 3% of tickets. Ship?

No. Six of the 200 cases hold a card number, so one miss is a 17% miss rate on those tickets. At 3,000 a day, 90 contain a card number and about 15 a day would leave the tenant. Widen the rule to any run of 13 to 19 digits whatever the spacing, add 20 card formats to the must-pass slice, and re-run until the count is zero.