Why we write these five by hand

Every rung of the ladder runs the same five mechanisms, whether its settings show them or not. A key store makes a side effect happen once. A deduper puts at-least-once deliveries back in order. A rate limiter answers "now, or when". A retry policy decides which failures earn another attempt. An approval gate holds an action until a person signs, on a clock. None of them calls a model, each fits in under 60 lines, and each fails at a boundary a default setting never mentions, which is why this module writes all five once, with tests.

A different company for this module

The agency from how to use this course sends 40 invoices a week, too few to stress them. This module follows a 40-person wholesaler that supplies 900 cafés. Its accounting system issues about 1,180 invoices a week and announces each one by webhook. An automation emails every invoice to the café, records payments, drafts credit notes for damaged deliveries, and at 16:00 on Friday sends 80 account statements. Every call goes through one API key whose bucket holds 10 requests and refills at one every 2 seconds. Staff time is the assumed $30 an hour and yours $100.

The version that trusts the defaults

The first build sat on rung 2 with every setting left as it came. One quarter's incidents, each a missing mechanism:

• No key store. In one week 1,240 deliveries covered 1,180 invoices; 60 were duplicates, and two reached cafés twice.
• No deduper. Eleven payment.received events arrived before their invoice, failed and were dropped; eleven cafés were chased for paid invoices.
• No rate limiter. The statement run fired 80 calls in one second; 70 got 429s and 69 failed for good. Resending them by hand took 2.3 hours, $69.
• A retry on everything. A record_payment call timed out after it had landed, and the retry recorded $4,200 twice. Untangling it cost $190.
• A gate with no rules. Three credit notes went out wrong: $380 approved two days late, $260 approved twice by a double tap, $540 approved by the agent that drafted it.

Assume each mechanism takes 2 hours of your time with its tests: 10 hours, $1,000 once. The gate's incidents alone, $380 + $260 + $540 = $1,180, cost more than that in one quarter, and the five functions then serve every automation the company runs.

Five mechanisms on one path

Expand

One path through the five mechanisms: webhooks enter the deduper, which hands gaps to a backfill; the key store passes only the first send per invoice and sends a changed amount to a review queue; the approval gate asks the founder to sign credit notes on a four-hour clock; the rate limiter grants a token; the retry policy sends the call to the accounting API100%−+FitSVG (opens in a new tab)CloseEsc

Pinch to zoom · drag to panScroll to zoom · drag or arrow keys to pan · 0 fits · Esc closes

The deduper stands at the door. It keeps a cursor, the next sequence number to apply, and holds an early arrival for at most 10 minutes, longer than any late delivery took last month. Anything below the cursor is a duplicate; a gap still open at 10 minutes goes to the backfill, which fetches it by number. A webhook deduper writes it.

The key store stands in front of the side effect. It remembers each invoice id for 72 hours from first sight, skips a repeat, and sends a changed amount under a live id to review rather than overwriting the first. Idempotency and deduplication explains why at-least-once delivery forces it; an idempotency key store writes it.

The approval gate holds every credit note over $200. Pending moves once, and the clock turns silence into expired, never approved. Approval gates decides where gates go; an approval-gate state machine writes this one.

The rate limiter is a copy of the vendor's bucket on your side, asked before every call. Its answer is a time: now, the second a token frees, or not within the wait the caller can afford. A token-bucket rate limiter writes it.

The retry policy wraps the single call. It classifies the error against the operation, waits with jitter, and stops at a 120-second budget. A retry policy writes it.

Why the order is the order

Deduper before key store. The key store keeps the first payload it sees under an id. Out of order, a corrected re-issue can land before the original, and the store keeps the correction and refuses the original. Ordering first makes "first seen" mean "first issued".

Gate before limiter. The limiter grants a token by reserving it. Take one and then wait four hours for a signature, and every call behind it waits on a reservation spent on nothing.

Retry inside the limiter. Every retry asks the limiter again, so retries never exceed the bucket. Skipping it is how 70 refused calls became about 210 more on Friday, three retries each into an empty bucket.

Key store outside the retry. The retry resends one call under one idempotency key; the key store stops a second decision to send, from a replayed run, a second webhook or the backfill.

What each looks like in a settings screen

In n8n, the rung-2 tool this course builds on, each mechanism is a setting that answers part of the question (Work with nodes (opens in a new tab)):

The history behind "value is new" holds 10,000 items by default, a window measured in items, not hours. At 1,240 deliveries a week it covers 10,000 ÷ 1,240 ≈ 8 weeks; it falls below a 72-hour span once volume passes 10,000 ÷ 3 ≈ 3,333 a day.

Why by hand, and not only configured

A setting covers the middle of the range; the five break at the edges, and each exercise puts its tests on one: a duplicate at exactly the TTL, a gap that never fills, a token that frees the second it is asked for, an approval at the deadline, a Retry-After longer than the budget. Write the test and the question for the vendor writes itself.

Above rung 2 nobody offers the settings. An agent SDK runs the model's loop and leaves the edges to you. llm-dev's a harness in 200 lines builds the inside of that loop; these five sit at its edges, and none belongs in a prompt, for the reason why reliability lives in the harness gives: "do not record a payment twice" cannot survive a lost response, and a key can.

In practice

Five questions, one per mechanism, for a vendor's demo or whoever built the workflow:

1. What is deduplication keyed on, for how long, and what happens to a changed payload?
2. Are deliveries applied in the sender's order, and what happens to one that never arrives?
3. Is the rate limit shared by every workflow that uses this API key?
4. Which errors are retried, and is a write with no idempotency key ever retried on a timeout?
5. When nobody answers an approval, what does it become, and who may approve it?

The answers you want are numbers and names: 72 hours, a 10-minute hold, one bucket per key, unkeyed writes retried on a 429 only, expired at 4 hours, anyone but the requester.

The mistake that costs money is switching on retry for a step that writes without a key. It looks like resilience and records the payment twice on the first timeout. The second is a deduplication history sized in items, which stops covering 72 hours once deliveries pass 3,333 a day.

Check yourself

1. Deliveries grow to 3,600 a day and the sender still retries for up to 48 hours. Is a 10,000-item history enough?

For the retries, yes: 3,600 × 2 = 7,200 items in 48 hours. For the 72-hour TTL, no: 10,000 ÷ 3,600 ≈ 2.8 days. Raise it above 3,600 × 3 = 10,800 with margin, or move to a store that expires by time, which volume cannot outgrow.

2. The statement run grows to 200, and the platform stops any execution at 5 minutes. Does one run finish?

No. Ten go at once, then one every 2 seconds, so the 200th leaves at 190 × 2 = 380 seconds. With max_wait at about 280 seconds the limiter grants 150 and refuses the last 50 when they ask. Split the run, 16:00 and 16:10, or ask the vendor for a larger bucket.