Writing tool descriptions the model can use

A tool description is documentation for a reader who cannot ask a question, cannot read the source and cannot try the tool to see what it does. The model has the name, the description and the argument schema, and from those three strings it decides which tool to call and with what. Every ambiguity you leave in the description becomes a wrong call, and wrong calls are measurable.

Most tool definitions we see in review were written by the engineer who wrote the function, in a hurry: search(query) with the description "Search." Accurate and useless, because it answers none of the three questions the model is asking.

The three questions a description answers

When do I use this? The situation, in the user's terms, not the implementation's. "Use when the user asks about an order they already placed" beats "Queries the orders table."

When do I not use this? The neighbour it is confused with. "Do not use for products the user has not bought; use search_catalogue for those." A model with twelve tools does not have a mental map of how they partition the space; it has twelve paragraphs, and the partition has to be written into them.

What does each argument mean? The unit, the format, the default, and the constraint the schema cannot express. "since: ISO 8601 date, inclusive, in the customer's timezone; omit for all time." The schema says string; only the description says which string.

Together those are a paragraph of 80 to 150 tokens per tool. Shorter than that and the model guesses; longer and the definition set outgrows the system prompt and is resent on every turn, as the tool-call loop showed.

The failure: twelve tools that overlap

A support assistant for a retailer shipped with twelve tools. Four of them touched orders:

get_order(order_id)            "Get an order."
lookup_order(email)            "Look up orders."
find_orders(query)             "Find orders matching a query."
order_status(order_id)         "Return the status of an order."

No description says which to prefer, what query accepts, or that order_status is a subset of get_order that costs a tenth as much. Asked "where is my order from last Tuesday", the model called find_orders("last Tuesday"), received nothing, called lookup_order with the user's name instead of their email, received an error, and gave up on the third turn with "I could not find your order."

That is a wrong-tool failure, and it has a rate. We wrote 50 realistic tasks, each with the tool call a competent agent would make first, and ran the assistant over them. It picked the wrong first tool on 11 of 50 and wrong arguments on a further 4: a 22% wrong-tool rate and 30% wrong-first-call rate before any answer was attempted. That measurement is the only honest way to review a tool surface, and it belongs in the golden set the evals module builds; a reviewer's opinion of a description is not evidence.

Expand

Three tool surfaces for one support assistant over 50 tasks: definition tokens, wrong first tool, wrong arguments, blast radius100%−+FitSVG (opens in a new tab)CloseEsc

Pinch to zoom · drag to panScroll to zoom · drag or arrow keys to pan · 0 fits · Esc closes

The fix: fewer tools, disjoint, each with a when-not

The second revision merged the four order tools into one and wrote the partition into every description:

get_order(order_id | email, since?)
  "Return the customer's orders with status, items and tracking.
   Use for anything about an order the customer has already placed.
   Pass order_id when the customer gives one; otherwise pass the
   account email and `since` (ISO 8601 date, inclusive) to narrow.
   Do not use for products not yet bought: use search_catalogue."

Twelve tools became seven. The definition set dropped from about 1,800 tokens to 1,100, and on the same 50 tasks the wrong first tool fell from 11 to 2 and wrong arguments from 4 to 2: four misses instead of fifteen, from a change to prose alone. The two remaining wrong-tool calls were on tasks where the user's intent was genuinely ambiguous, and the fix for those is a clarifying question, not another paragraph.

Three rules fell out of the rewrite:

• One tool per verb, not one per table. The model thinks in what the user wants, and so should the surface.
• Name the neighbour. Every "do not use for X" names the tool that does X. A model told what not to do with no alternative does it anyway.
• Put constraints in the schema where the schema can hold them, and in the description where it cannot. enum for the five statuses; prose for "inclusive".

One shell tool versus twelve narrow ones

The opposite design is tempting: give the model one tool, run(command), and let it compose. The wrong-tool rate goes to zero by construction. The description is 200 tokens instead of 1,100, and nothing needs rewriting when the API changes.

Over the same 50 tasks it did worse. Nine wrong first calls, all in the arguments: a wrong flag, a wrong path, a query that returned 40 KB where 400 bytes was wanted. And every one of those 40 KB results entered the context, which memory and context management will price. The model is good at choosing between seven well-separated intentions and worse at writing shell from a description of a host it cannot see.

The deciding argument is not accuracy, though. A shell tool's blast radius is the host: whatever the process can read, the model can read, and whatever it can run, a page it fetched can ask it to run. The least-privilege lesson makes this concrete; here the short version is that a narrow tool is a permission boundary and a shell tool is the absence of one. Use a shell tool in a sandbox, for a developer, on a task where composing is the point. Use narrow tools for anything that touches a customer's data.

The same question comes up when the surface is a server other people's clients will load, and the founder-facing course works it for a company's own systems in designing the tool surface.

Reviewing a tool surface

Before the tool set ships, do what we did above, and keep the artefacts:

1. Write 50 tasks in the user's words, with the expected first call and arguments.
2. Run the assistant once per task and record the first call.
3. Report wrong tool and wrong arguments separately; they have different fixes.
4. For each miss, decide whether the fix is a description, a schema constraint, a merge, or a clarifying question.
5. Rerun. The number has to move.

Fifty tasks at one call each is about 75k input tokens. At the course's latency figures (about 1 s to the first token and 50 tokens/s, from the four numbers) a first call of about 60 tokens takes 2.2 s, so the run takes under two minutes even one task at a time. A surface that was never measured this way has an unknown wrong-tool rate.

In an interview

The interviewer is checking whether you treat the description as an interface with a measurable failure rate, or as a comment. Say the three questions, show a before-and-after description, and give the number: wrong first call on 50 tasks, before and after. Then take the shell-tool question head on: zero wrong-tool by construction, more wrong-argument, and no permission boundary.

The mistake that loses points is describing what the function does instead of when to call it. "Queries the orders table by id" is the implementation; the model needed "use when the customer asks about an order they placed; otherwise use search_catalogue". A close second is answering "how would you improve the tools" with vocabulary rather than with a 50-task measurement.

Check yourself

1. The assistant has search_docs(query) and search_tickets(query), and 8 of 50 tasks call the wrong one. Which fix do you try first, and how do you know it worked?

Rewrite both descriptions with a when-not that names the other ("do not use for a customer's own case history; use search_tickets"), because the schema is identical and only the prose can partition them. Rerun the 50 tasks; the 8 must fall to 2 or fewer, and the ones that remain should be tasks where the user's words could mean either, which a clarifying question handles.

2. The definition set is 4,000 tokens across 25 tools, and a typical task is 4 turns. What does the set cost per task, and when is it worth the merge work?

4,000 × 4 = 16,000 input tokens per task on definitions alone, at P_in, or at C_r × P_in if the set is byte-stable and cached. Merging to 10 tools at 1,500 tokens saves 10,000 tokens per task. Worth it when either the wrong-tool rate is above a few percent, or the monthly saving, tasks per day × 30 × 10,000 tokens × P_in, exceeds the engineer time the merge takes; both are usually true past a few thousand tasks a day.

3. A teammate proposes replacing the seven tools with run(sql) against a read replica, arguing the model "can express anything". Accept or refuse?

Refuse for the customer-facing path: the wrong-argument rate rises, a bad query returns tens of kilobytes into context, and the permission boundary becomes "whatever the replica user can read", which an injected instruction can exploit. Accept it behind a sandbox for an internal analyst tool where composing queries is the task and the data is already theirs.