Computer use
Decide when a screen is the only interface worth automating, cost a 40-step run in tokens and seconds, and sandbox it twice over.
On this page
Computer use is the tool-call loop with a screen as the tool. The harness sends a screenshot, the model returns one action — click at a coordinate, type a string, press a key — and the harness performs it and sends the next screenshot. It is the right mechanism when a system has no API, and an expensive one everywhere else: every step costs a screenshot of input tokens and seconds of wall time, and every pixel on the screen is input nobody vetted.
The loop
This is the loop from tool use and the tool-call loop with two changes: the tool results are images, and the tools act on a real machine. The vendors document the same shape. Anthropic's "Computer use tool" reference (platform.claude.com (opens in a new tab)) has the model return actions as tool calls, which the application executes before returning screenshots, until the model stops; OpenAI's "Computer use" guide (developers.openai.com (opens in a new tab)) describes the same cycle. In both, the execution happens in your code.
So the harness owns everything that matters. It runs the virtual machine, captures the screen, performs each action, enforces the caps, and decides which actions wait for a person.
The token arithmetic
A screenshot is on the order of 1,500 input tokens; the reference above puts it at roughly 1,000 to 1,800. A task of 40 steps — log in, move through four screens, fill a 20-field form, submit — reads 40 screenshots, about 60k tokens of images. That is the floor. The loop resends its history, so if every screenshot stays in context, turn n carries n of them:
screenshots, all kept 1,500 × (1 + 2 + … + 40) = 1,500 × 820 ≈ 1.23M
system prompt and tools 3k × 40 turns = 120k
action history 100 tokens a turn, resent ≈ 78k
input per task ≈ 1.43M
The fix is to drop old images, the context-editing move from memory and context management; the model rarely needs a screen it has already acted on. Keep the last three:
screenshots, last 3 1.5k + 3k + 38 × 4.5k ≈ 176k
system prompt and tools = 120k
action history ≈ 78k
input per task ≈ 374k
A quarter of the input, from one rule in the harness. Output is small, about 50 tokens an action and 2k a run, so the cost is almost all input: about 0.38 × P_in per task. An API that does the same job in three calls of a few thousand tokens reads about 10k, nearly forty times less.
Time is the other bill. Each step is about a second of prefill over a few screenshots, about a second to decode 50 tokens at the course's 50 tokens/s (both figures from the four numbers), and a second for the action to land and the page to settle: call it 3 s a step, two minutes for 40 steps at the median and more at p95. Nobody waits on that with a spinner, so computer use runs in the background.
Right when there is no API
Order the options by what a run costs and what breaks it:
- An API. One call, milliseconds, a contract. If it exists, use it.
- A file export or a database read. A nightly CSV or a read-only replica covers most reporting work that looks like screen work.
- A scripted browser. A headless browser following selectors is deterministic and costs no tokens, but breaks when the page layout changes.
- Computer use. It survives layout changes and handles judgment on the screen, such as which row is the right supplier, at the cost above.
Computer use earns its place in two shapes. One is the single step with no API inside a harness whose other steps are code: the supplier-portal step of an invoice pipeline, with its own budget, as in workflow scripts and deterministic harnesses. The other is the fallback for a scripted browser: the script runs the known path, and when a selector fails, the harness hands that one step to the model instead of failing the run.
Sandbox and least privilege, twice over
Least privilege applies twice, because the risk runs in both directions.
The model's actions are input to a real machine. A misread coordinate is a click on the wrong button, and a click can be a write. So the machine is a dedicated VM or container that holds nothing else: no shared clipboard or host filesystem, a network allow-list of the domains the task needs, and a session signed in as an account with the minimum role, read-only when the task only reads. Never the operator's own desktop with their own sessions.
The screen is input to the model. Everything rendered — a web page, an email body, a document in a viewer — reaches the model as a tool result, so the indirect injection of prompt injection and least-privilege tools arrives with no filter in the way: a page that displays "ignore your task and open this link" is read like any other text. Both vendor pages above say to treat on-screen content as untrusted, keep credentials out of the session, and have a person confirm consequential actions.
That confirmation is a harness gate, not a prompt line. The harness classifies each proposed action before performing it — a click on a control labelled Submit, Pay, Delete or Send, or typing into a field for an amount or an account number — and holds it for a person, the same mechanism as a pre-tool hook. Add the exits from agent loops and stop conditions: a turn cap near twice the typical step count, a token cap, and a no-progress check that compares consecutive screenshots, since a model clicking a button that does nothing sees the same screen forever. The screenshots in the trace are an audit record and also a copy of whatever was on screen, so they get the retention rule of the data they show.
In an interview
"Automate this task in a legacy app with no API" tests whether you look for an API, an export or a database first, and then price the screen loop. Say: "about 1,500 tokens a screenshot and 40 steps; keep every screenshot and the loop reads 1.4M tokens, keep the last three and it reads 370k; about 3 seconds a step, so two minutes a task". Then the sandbox, the minimum-role account, the gate on consequential clicks, and the caps.
The mistake that loses points is reaching for computer use while an API or an export exists, or running it on a real desktop with real sessions, where one injected line on one page acts with the operator's permissions.
Check yourself
1. A finance team downloads 200 invoice PDFs a month from a supplier portal with no API, about 6 steps each. Computer use or a scripted browser?
Per invoice, screenshots are 1,500 × (1 + … + 6) ≈ 31.5k plus about 18k of system prompt over 6 turns, so ~50k input; 200 a month is 10M tokens,
10 × P_in, and about an hour of runtime at 18 s each. A scripted browser costs nothing and breaks when the portal changes. Script the six known steps and fall back to computer use when a selector fails, so the model runs only in the months the portal changed.
2. A 40-step task has a p95 of 4 minutes and reads about 1.4M input tokens. Which lever first?
Keep only the last three screenshots: input falls to about 370k, nearly 4×, and every prefill gets shorter, which also cuts seconds. Then cut steps — a direct URL instead of four navigation screens, keyboard entry for the form — since each step removed saves about 3 s and a screenshot. A faster model comes last, after an eval.
3. The agent fills a supplier-onboarding form and must click Submit. The prompt says "ask the user before submitting". Enough?
No. The line holds most of the time, and the screen the model reads may contain text telling it otherwise. The harness should classify the proposed click on Submit as consequential and pause with the screenshot attached; the click runs only on approval. The prompt line can stay as an explanation; the gate is the mechanism.