Prompt and context engineering on real data · Context assembly
Chunk the policy manual
Split the policy wording into chunks a retriever can cite by clause number, capped at 400 tokens, without mistaking a sentence for a heading.
codeeasyExercise · 3 min · 36 of 64
On this page
The gold set scores the pilot by clause number, and the prototype's chunker made that impossible. Its replacement is about forty lines, and the line that matters is the one deciding what counts as a heading.
The insurer's 340-page policy set, the motor, property and commercial wordings plus their endorsements, arrives as text extracted from the PDFs: one string per printed line, cover and contents pages removed. Sections are numbered and titled, 4 Exclusions, 4.2 Escape of water, 4.2.1 Frozen pipes. The prototype split on any line beginning with a digit and kept each section whole, and three things broke. Body lines such as "14 days after the leak is found" became sections of their own, 61 of them. Commercial sections of up to 2,700 words, about 3,600 tokens, went into the prompt whole, three times the 1,200-token clause band from context assembly. And chunks were numbered by position, so the QA lead's 200 gold labels, written as clause numbers, had to be matched by hand. The retriever puts three chunks in each prompt, so the cap is 400 tokens. Assume 0.75 words per token: a chunk's tokens are its words ÷ 0.75, rounded up.
Implement solve so that every chunk is citable by the clause number the handlers use, never mixes two sections, and never exceeds the cap unless a single paragraph does.
def solve(lines, max_tokens):
"""Split one policy document into chunks a retriever can cite.
lines one string per printed line; the first line is a heading.
A line is a heading when it is a section number, a space and a
title starting with a capital letter ('4.2 Escape of water'), AND
the number is one the document can have next: the first child of
the current section (4.2 -> 4.2.1), or the next sibling of the
current section or of one of its ancestors (4.2 -> 4.3 or 5).
Any other line is body text. A blank line ends a paragraph; a
wrapped line continues it.
max_tokens the cap, counting body words at 0.75 words per token, rounded up.
Returns [(chunk_id, heading_path, text, tokens), ...] in document order:
chunk_id the section number, '4.2'; when a section needs more than
one chunk, '4.2#1', '4.2#2', ...
heading_path titles from the top-level section down, joined by ' > '
text the chunk's paragraphs, lines joined by single spaces,
paragraphs separated by newlines
tokens the chunk's body tokens
Consecutive paragraphs share a chunk while it fits the cap; a paragraph
over the cap is a chunk on its own and is never cut. A heading with no
body text of its own produces no chunk.
"""
lines = [
"4 Exclusions",
"4.1 Wear and tear",
"We do not cover gradual deterioration.",
"4.2 Escape of water",
"We cover sudden escape of water",
"from a fixed installation.",
"",
"14 days after the leak is found, cover ends",
"unless the leak has been reported to us.",
"5 Claims",
"5.1 Reporting",
"Tell us within 30 days.",
]
solve(lines, 400) == [
("4.1", "Exclusions > Wear and tear", "We do not cover gradual deterioration.", 8),
("4.2", "Exclusions > Escape of water",
"We cover sudden escape of water from a fixed installation.\n"
"14 days after the leak is found, cover ends unless the leak has been reported to us.", 36),
("5.1", "Claims > Reporting", "Tell us within 30 days.", 7),
]
# At a 20-token cap, 4.2 becomes '4.2#1' (14 tokens) and '4.2#2' (23 tokens):
# the second paragraph is over the cap and stays whole.
A strong answer · Rubric · Where answers go wrong · Solution
Two passes. The first reads lines into sections. A heading-shaped line is a heading only if its number can come next, so the current number is kept as a tuple of integers and compared against its first child and the next sibling at each level. Under 4.2 the candidates are 4.2.1, 4.3 and 5, which makes "14 days after the leak" body text; "2 or more occupants" fails on its lower-case title even where 2 would be a valid next number. Tuples rather than strings also put 4.10 after 4.9. Lines join the open paragraph, a blank line closes it, and the heading path is the titles of the number's prefixes.
The second pass packs each section's paragraphs greedily: add the next paragraph while the chunk's tokens, words × 4 ÷ 3 rounded up, stay within the cap; otherwise close the chunk and start another. A paragraph over the cap is emitted whole, because a clause cut mid-sentence leaves its condition in one chunk and its exception in the next. A section in one chunk keeps the bare number; a split one gets #1, #2, so the gold label "4.2" still matches on the part before the #. A parent heading with no body produces nothing, and an empty document returns an empty list.
What it never does is cross a section boundary or number chunks by position. The rule will occasionally reject a real heading, a running page header or a skipped number, and the check that catches it is cheap: the count of sections found against the contents page the extractor removed.
"Split on any line that starts with a number." That is the prototype. "14 days after the leak is found" becomes section 14, its sentence is cut off from 4.2, and the retriever returns half an exclusion with no subject.
"Cut every 400 tokens with an overlap, and keep the page number." A fixed window crosses section boundaries: a chunk holding the end of 4.2 and the start of 4.3 can be cited as neither, and the gold set's clause numbers have nothing to be equal to.
"Cut a long paragraph at the cap." A 600-token exclusion split mid-sentence puts the exclusion in one chunk and its exception in another, and retrieving only the first returns the opposite of what the clause says.
Python 3 · standard library101 lines · 7 of 7 tests passing
"""Chunk the policy manual — split on numbered sections, cap each chunk, cite by clause number."""
import re
# A heading-shaped line: a dotted section number, a space, a title. Whether it is
# a heading is decided by the sequence rule and the capital letter, not the shape.
HEADING = re.compile(r"^(\d+(?:\.\d+)*)\s+(\S.*)$")
def tokens(words):
"""Tokens at the assumed 0.75 words per token, rounded up: ceil(4w / 3)."""
return (4 * words + 2) // 3
def is_next(number, current):
"""True when `number` can follow `current`: its first child, or the next
sibling of it or of one of its ancestors. Under 4.2 that is 4.2.1, 4.3 or 5;
a body line that starts '14 days after...' is none of them."""
if number == current + (1,):
return True
return any(number == current[:d] + (current[d] + 1,) for d in range(len(current)))
def pack(paragraphs, max_tokens):
"""Consecutive paragraphs share a chunk while it fits the cap. A paragraph
over the cap alone becomes a chunk alone: it is never cut mid-sentence."""
chunks, current, words = [], [], 0
for para in paragraphs:
n = len(para.split())
if current and tokens(words + n) > max_tokens:
chunks.append((current, words))
current, words = [], 0
current.append(para)
words += n
if current:
chunks.append((current, words))
return chunks
def read_sections(lines):
"""[number, heading_path, paragraphs] per section, in document order."""
sections = []
titles = {} # section number -> title, for the heading path
current = None # number of the section being read
para = [] # lines of the paragraph being read
def end_paragraph():
if para and sections:
sections[-1][2].append(" ".join(" ".join(para).split()))
para.clear()
for raw in lines:
line = raw.strip()
m = HEADING.match(line)
if m:
number = tuple(int(part) for part in m.group(1).split("."))
title = m.group(2)
if title[0].isupper() and (current is None or is_next(number, current)):
end_paragraph()
current = number
titles[number] = title
path = " > ".join(titles[number[:i]] for i in range(1, len(number) + 1)
if number[:i] in titles)
sections.append([number, path, []])
continue
if line:
para.append(line) # a wrapped line continues the paragraph
else:
end_paragraph() # a blank line ends it
end_paragraph()
return sections
def solve(lines, max_tokens):
out = []
for number, path, paragraphs in read_sections(lines):
if not paragraphs:
continue # a parent heading with no body text of its own
clause = ".".join(str(n) for n in number)
chunks = pack(paragraphs, max_tokens)
for i, (paras, words) in enumerate(chunks, 1):
# The id is the clause number the handlers cite, so a gold label
# scores by equality; a split section keeps it, with a suffix.
chunk_id = clause if len(chunks) == 1 else f"{clause}#{i}"
out.append((chunk_id, path, "\n".join(paras), tokens(words)))
return out
EXAMPLE = [
"4 Exclusions",
"4.1 Wear and tear",
"We do not cover gradual deterioration.",
"4.2 Escape of water",
"We cover sudden escape of water",
"from a fixed installation.",
"",
"14 days after the leak is found, cover ends",
"unless the leak has been reported to us.",
"5 Claims",
"5.1 Reporting",
"Tell us within 30 days.",
]
The cases that ran
TESTS = [
# The page example: '14 days...' is body text, and the parent headings 4 and 5 have no chunk.
((EXAMPLE, 400),
[("4.1", "Exclusions > Wear and tear", "We do not cover gradual deterioration.", 8),
("4.2", "Exclusions > Escape of water",
"We cover sudden escape of water from a fixed installation.\n"
"14 days after the leak is found, cover ends unless the leak has been reported to us.", 36),
("5.1", "Claims > Reporting", "Tell us within 30 days.", 7)]),
# The same document at a 20-token cap: 4.2 splits on its paragraph break, and the
# second paragraph stays whole at 23 tokens rather than being cut.
((EXAMPLE, 20),
[("4.1", "Exclusions > Wear and tear", "We do not cover gradual deterioration.", 8),
("4.2#1", "Exclusions > Escape of water", "We cover sudden escape of water from a fixed installation.", 14),
("4.2#2", "Exclusions > Escape of water",
"14 days after the leak is found, cover ends unless the leak has been reported to us.", 23),
("5.1", "Claims > Reporting", "Tell us within 30 days.", 7)]),
# '2 or more...' is the next number the document could have, but its title is not
# capitalised, so it is body text; 1.1.1 carries a three-level heading path.
((["1 Cover",
"1.1 Your vehicle",
"We cover your vehicle while it is parked or driven.",
"2 or more occupants must wear seat belts.",
"1.1.1 Keys",
"Keys left in the vehicle void theft cover.",
"2 Excess",
"The first 250 of each claim is yours."], 400),
[("1.1", "Cover > Your vehicle",
"We cover your vehicle while it is parked or driven. 2 or more occupants must wear seat belts.", 24),
("1.1.1", "Cover > Your vehicle > Keys", "Keys left in the vehicle void theft cover.", 11),
("2", "Excess", "The first 250 of each claim is yours.", 11)]),
# Packing: the first two paragraphs fit an 8-token cap together, the third does not.
# The first heading may carry any number.
((["3 Theft", "Theft is covered.", "", "Attempted theft too.", "", "Theft by family is not covered."], 8),
[("3#1", "Theft", "Theft is covered.\nAttempted theft too.", 8),
("3#2", "Theft", "Theft by family is not covered.", 8)]),
# Stray whitespace and repeated blank lines change nothing; a trailing heading with no body has no chunk.
(([" 2 Excess ", "", " The first 250 of each claim", "is yours. ", "", "", "3 Theft"], 400),
[("2", "Excess", "The first 250 of each claim is yours.", 11)]),
# Numbers compare as integer tuples: 4.10 follows 4.9, and a later '4.2 ...' line is body text.
((["4.9 Storm", "Wind damage is covered.", "4.10 Flood", "Rising water is covered.", "4.2 Theft is not."], 400),
[("4.9", "Storm", "Wind damage is covered.", 6),
("4.10", "Flood", "Rising water is covered. 4.2 Theft is not.", 11)]),
# An empty document.
(([], 400), []),
]
