Implement the parser that turns a prose reply into calls to dispatch and errors to send back, and never executes a brace from the prose.
codeeasyExercise · 4 min · 68 of 77
On this page
A model with no native tool-call field writes its calls as text, and the harness has to find them. Every reply passes through the parser first, so a bug in it is either a crashed run or a call nobody asked for.
A team runs its support agent on a self-hosted model with no structured tool-call field, so the system prompt asks for each call as a fenced json block holding name and args. The first parser was a regular expression for the outermost pair of braces, handed to json.loads. Over a week of 14,000 replies it failed three ways. 120 replies (0.9%) carried a trailing comma or a single-quoted key; json.loads raised, and the run died with no trace. 40 replies were cut off at the output cap in the middle of a block. One reply described a call it had decided against, quoted {"name": "delete_draft"} in a sentence, and the harness executed it.
The harness wants one function that turns a reply into the calls to dispatch and the errors to send back as results, so that a bad block costs one round trip instead of a run, as the tool-call loop requires of every failure.
Implement the parser so that every malformed block becomes an error entry naming its block, and nothing outside a json fence is ever parsed.
def solve(text):
"""text: one model reply. A tool call is a fenced block whose opening line
is three backticks and json, in any case, holding one object
{"name": str, "args": {...}}; "args" may be omitted. Fences with any
other tag are prose.
Returns one entry per json block, in order:
("call", name, args) a well-formed call
("error", n, reason) the n-th json block (from 1) was malformed;
reason is "invalid json", "not an object",
"missing name", "args not an object" or
"unclosed block"
Prose, and braces in prose, are never parsed. Never raises."""
Walk the reply line by line with three states: prose, inside a json block, and inside any other fence. The third state is the one most answers miss. Without it, a Python sample's closing fence reads as the opening of a new block, and everything after it is parsed out of step. An opening line of three backticks and json starts a block and increments the block counter; a closing fence hands the collected lines to json.loads.
Check the shape only after the JSON parses, in a fixed order: an object, then a non-empty string name, then args as an object, defaulting to an empty one when omitted. Each failure is an entry ("error", n, reason) in the same list as the calls, in reply order, so the harness can return "block 2: invalid json" as a result while it dispatches block 1. A block still open when the text ends is reported as unclosed block and not parsed. That is a reply cut at the output cap, the stop reason reading the response checks first, and half an argument list on send_reply is a message sent with half a body.
The function never raises, never scans prose for braces, and never repairs. Closing a truncated string is a guess at what the model meant, and the call it produces may be a write. The error costs one round trip and the model re-emits the block. One pass, linear in the length of the reply, about thirty lines; it is the first stage of the loop in a harness in 200 lines.
"Find the first { and the last } and parse what is between them." That is the regex that executed delete_draft from a sentence. With two blocks in one reply it also joins both into one string that never parses, so a reply with one good call and one bad one dispatches neither.
"Wrap the whole parse in try and return an empty list on any error." An empty list is what a reply with no calls looks like, so the loop reads a malformed call as the model's final answer and ends the run half done, recorded as done.
"Run a repair pass that closes open strings and braces before parsing." It turns the 40 truncated replies into 40 calls with invented arguments. The error result costs one round trip; a repaired send_reply can cost a message to a customer that cannot be unsent.
Python 3 · standard library65 lines · 12 of 12 tests passing
"""JSON tool-call parser — pull fenced tool calls out of a model's prose reply."""
import json
FENCE = "`" * 3 # built rather than typed, so this file can be quoted inside a markdown fence
def block(body, tag="json"):
"""A fenced block as the model writes one. The tests build their replies with it."""
return f"{FENCE}{tag}\n{body}\n{FENCE}"
def shape_error(obj):
"""The shape rules, in the order they are checked. None means a well-formed call."""
if not isinstance(obj, dict):
return "not an object"
name = obj.get("name")
if not isinstance(name, str) or not name:
return "missing name"
if not isinstance(obj.get("args", {}), dict):
return "args not an object"
return None
def parse(n, body):
"""One closed json block, numbered n, to a call or an error entry. Never repairs."""
try:
obj = json.loads(body)
except (ValueError, RecursionError): # JSONDecodeError is a ValueError; nesting past the stack limit is not
return ("error", n, "invalid json")
reason = shape_error(obj)
if reason:
return ("error", n, reason)
return ("call", obj["name"], obj.get("args", {}))
def solve(text):
"""text: one model reply. A tool call is a fenced block whose opening line
is three backticks and json, in any case, holding one object
{"name": str, "args": {...}}; "args" may be omitted. Fences with any
other tag are prose.
Returns one entry per json block, in order:
("call", name, args) a well-formed call
("error", n, reason) the n-th json block (from 1) was malformed;
reason is "invalid json", "not an object",
"missing name", "args not an object" or
"unclosed block"
Prose, and braces in prose, are never parsed. Never raises."""
out, buf, n = [], [], 0
state = "prose" # prose | call (inside a json fence) | other (any other fence)
for line in text.split("\n"):
mark = line.strip()
if state == "prose":
if mark.lower() == FENCE + "json":
state, buf, n = "call", [], n + 1
elif mark.startswith(FENCE):
state = "other" # a code sample: its contents are never calls
elif mark == FENCE:
if state == "call":
out.append(parse(n, "\n".join(buf)))
state = "prose" # invariant: every opened block is closed or reported below
elif state == "call":
buf.append(line)
if state == "call":
out.append(("error", n, "unclosed block")) # the reply stopped mid-call, usually at the output cap
return out
The cases that ran
TESTS = [
# the examples on the page
(("Let me check the queue.\n" + block('{"name": "search_tickets", "args": {"q": "refund delay"}}'),),
[("call", "search_tickets", {"q": "refund delay"})]),
((block('{"name": "get_order", "args": {"id": "A-17"}}') + "\nthen\n"
+ block('{"name": "get_shipment", "args": {"id": "A-17"},}'),),
[("call", "get_order", {"id": "A-17"}), ("error", 2, "invalid json")]),
(("Drafting the reply.\n" + FENCE + 'json\n{"name": "send_reply", "args": {"body": "Hi Priya',),
[("error", 1, "unclosed block")]),
((block("print({'a': 1})", "python") + "\nNow the tests.\n" + block('{"name": "run_tests"}'),),
[("call", "run_tests", {})]),
# braces in prose are never a call: the incident that started this
(('An example call looks like {"name": "delete_draft", "args": {}} in the docs.',), []),
(("",), []),
((block('["search_tickets", {"q": "x"}]'),), [("error", 1, "not an object")]),
((block('{"args": {"q": "x"}}'),), [("error", 1, "missing name")]),
((block('{"name": "search_tickets", "args": "refund"}'),), [("error", 1, "args not an object")]),
((block("", "json"),), [("error", 1, "invalid json")]),
# nesting deeper than the parser's stack is an error entry, not a crash
((block("[" * 100000),), [("error", 1, "invalid json")]),
# numbering counts json blocks only, and a bad block does not stop the next one
((block("print(1)", "python") + "\n" + block("{name: 'a'}") + "\n" + block('{"name": "a"}', "JSON"),),
[("error", 1, "invalid json"), ("call", "a", {})]),
]
