Idempotency and deduplication
Trace a duplicate invoice to at-least-once delivery, choose the idempotency key and where it lives, and size its expiry from the sender's retry window.
On this page
A webhook resent after a timeout, a POST retried after a dropped connection, a poll window that overlaps the last one: each delivers the same change twice, and each does it by design. At-least-once delivery means duplicates, and duplicates mean idempotency is required. For a connector, duplicates are normal traffic, not an incident, and the receiver's job is to make the second copy harmless.
The month of double invoices
The agency moved the Friday invoice run from the arithmetic of a task onto a webhook. When a project's week is approved, the time tracker POSTs an event. The handler creates the invoice in the accounting system, asks the model to turn the timesheet notes into line descriptions, and places the invoice in the operations manager's review queue. All of it ran inside the handler, before the reply. Most deliveries finished in 8 seconds, and on slow days the model call pushed some past 12. The time tracker waits 10 seconds for a 2xx, an assumption here since providers range from a few seconds to about 30, and then resends.
The first month had 173 invoices (40 a week × 4.33). Eleven deliveries ran past 10 seconds, all 11 were resent, and each resend created a duplicate invoice. The review compares hours, rate and total, and a duplicate passes all three because its numbers are correct. Seven were caught only because the operations manager saw two identical invoices in the queue. Four reached clients, and one client paid both.
void + credit note 11 × 15 min × $30 = $82.50
founder apology 4 × 30 min × $100 = $200.00
refund the double pay 1 × 45 min × $30 = $22.50
≈ $305 in the month
The automation was priced to save $412 a month. Duplicates took $305 of it. The cost that is not in the sum is four clients who now check every invoice the agency sends.
Why the sender is right to resend
From the time tracker's side, a delivery with no reply in 10 seconds has two explanations it cannot tell apart: the request never arrived, or it arrived and the reply was lost or late. Not resending loses invoices, which is at-most-once. Resending duplicates some, which is at-least-once. Every webhook provider chooses at-least-once, and so does every queue and every sensible retry loop. At-least-once delivery means duplicates; duplicates mean idempotency is required. Exactly-once delivery is not on sale. An exactly-once effect is, and the receiver builds it. The general derivation, with the payments version, is idempotency, and why at-least-once forces it.
Answering first and working later, as webhooks versus polling recommends, cut the agency's timeouts to nearly zero. It did not remove duplicates. The time tracker still resends after a network error, after its own crash, or after a deploy on the agency's side drops a connection mid-reply. Duplicates became rarer, which made each one more surprising.
Two keys: one inbound, one outbound
A connector needs a key in each direction.
Inbound: the event id. Each webhook carries an id that stays the same across resends of that event. Record it before acting; a later delivery with a recorded id is answered 200 and dropped. Check which id is stable. Some providers send a delivery id, new on every attempt, alongside the event id, and deduping on the delivery id dedupes nothing. Where no stable id exists, derive one from what identifies the change: here, the project and the week.
Outbound: the key on the write. When the connector creates the invoice, it sends a key for the logical operation, ts-881 for timesheet 881, on every attempt. If the accounting API honours an Idempotency-Key header, a repeated create returns the first invoice instead of making a second. If it does not, use a natural key: write the timesheet id into the invoice's reference field and search for it before creating. The search has a race, since two workers can both find nothing and both create, so it needs one worker per timesheet or a lock in your own key store.
The inbound key stops a duplicate event. The outbound key stops a duplicate effect when your own handler crashes between creating the invoice and recording that it did. You need both, because the gap between two writes in two systems, your key store and their invoice, is exactly where a crash leaves one without the other.
Where the key lives
Keep the keys in a table with a unique constraint on the key: insert first, and let the constraint refuse the second insert. The SQL is in the system-design lesson above, and why the lookup costs the same at 300 rows or 3 million is what a hash map buys. Never keep them in the worker's memory. A deploy restarts the worker and empties the set, and a deploy is exactly when in-flight deliveries fail and come back.
Each row holds the key, when it was first seen, a hash of the payload and the result, here the invoice number. Three rules follow:
- A repeat with the same payload returns the stored result, and nothing is created.
- A repeat with a different payload, such as a timesheet re-approved with corrected hours under the same event, is neither processed nor allowed to overwrite. It goes to a person. Replacing the first invoice silently is how a client ends up with two different invoices under one number.
- The first-seen time is never refreshed by a duplicate. Otherwise a sender stuck resending keeps its own key alive for as long as it keeps resending.
How long to keep it
Long enough to outlast the sender's retries, with margin. If the time tracker resends for up to 48 hours, keep keys for 72. Shorter, and a resend at hour 50 finds no key and creates a duplicate. Longer costs little: at 40 invoices a week, 72 hours of keys is at most 40 rows, and even 10,000 events a day kept for 72 hours is 30,000 rows of a few hundred bytes, about 10 MB. The limit runs the other way: a legitimate re-issue under the same id after the window must get through, and a store that never expires blocks it forever. An idempotency key store builds this store, with the three rules above and a test at exactly the expiry.
Deduplication is not idempotency
Deduplication drops a repeated message. Idempotency makes repeating an operation harmless. The first is a filter at the door; the second is a property of the write. A connector needs both, because the filter only catches duplicates it can recognise. An incremental poll's overlap re-delivers records with no event id at all, and only an idempotent write makes them harmless: upsert by the record's own id, applied only if its version is newer than the stored one. The same holds behind a message queue, which delivers at least once and hands every consumer the same problem.
In practice
Ask the vendor: does the event id stay the same across resends, and for how long do you resend? Do your write endpoints accept an idempotency key, and how long do you remember it? Is there a reference field we can search? For a connector you buy, ask where it records processed ids, on which id, and for how long.
The mistake that costs money is deduplicating in memory, or on an id that changes with every attempt. Both pass every test, because tests do not restart the worker or resend with a fresh delivery id, and both double-invoice on the first deploy or the first resend in production.
Check yourself
1. The provider resends webhooks for up to 3 days. Your key store expires keys after 24 hours. Decide the expiry, and say what the current one costs.
Any event processed but unacknowledged, for example during an outage on your side, can come back between hour 24 and hour 72, find no key, and run again. Set the expiry to 96 hours: the 72-hour window plus a day of margin. At the agency's volume that is still at most 40 rows, one week's invoices.
2. For the invoice event, choose the key: the delivery id, the event id, or a hash of project, week and amount.
The event id, if the provider keeps it stable across resends. Otherwise project and week, without the amount. With the amount in the key, a corrected re-issue looks like a new invoice and gets created. Without it, the corrected re-issue arrives under a live key with a different payload, and goes to a person.
3. The accounting API has no idempotency key, so the connector searches by reference before each create. Is one extra request per invoice worth it under a 30-a-minute limit?
It is 173 extra requests a month, under six minutes of the limit spread over a month. One avoided duplicate saves at least $7.50 of voiding and possibly $50 of the founder's apology. Yes, and it is the only defence left when the handler crashes between the create and the record.