Idempotency, and why at-least-once forces it

A charge commits in the database and the response is lost on the way back. The client cannot tell a lost response from a lost request, so it sends the request again, and the customer pays twice. Idempotency is the property that makes that second attempt free: running the operation once or five times leaves the system in the same state.

The chain that forces it

Networks lose responses. A timeout, a reset connection, a load balancer returning 502 after the app server committed — from the caller's side these are all one observation, and it is ambiguous. Two worlds produce it: the request never arrived (zero effects), or the reply was lost (one effect). No amount of waiting separates them; this is the Two Generals problem, and it has no solution.

The caller therefore has exactly two policies. Never retry, which is at-most-once and silently drops writes. Or retry, which is at-least-once and produces duplicates. Anyone handling money picks retry, so duplicates stop being a failure mode and become normal traffic.

Put a number on "normal". A payments endpoint taking 1,000,000 charge requests/day runs at 1,000,000 ÷ 86,400 ≈ 12 QPS average, and at a 4x peak roughly 50 QPS. Assume 0.1% of calls end ambiguously; a Mumbai client calling US East pays ~200 ms per round trip against a 2 s timeout, and deploys and GC pauses add to that. So 1,000,000 × 0.001 = 1,000 ambiguous outcomes per day, every one of them retried. If the effect had already committed, each retry is a second charge. At an assumed ₹800 average ticket, that is about ₹800,000 a day in refunds and chargebacks, produced by correct client behaviour.

The queue side is worse. A consumer with a 30 s visibility timeout and a handler whose p99 is 35 s has its slowest 1% redelivered while still running: 1,000,000 × 0.01 = 10,000 duplicate deliveries a day, before a single crash. Retries live in more than one place:

Where duplicates come from: retries at the client, the gateway, and the queue100%−+FitSVGEsc

Scroll to zoom · drag to pan · 0 fits · Esc closes

So: lost responses ⟹ retries ⟹ at-least-once ⟹ duplicates are guaranteed, not possible ⟹ every non-idempotent operation must be made idempotent, or it double-charges.

Which operations are already safe

Some operations survive replay for free. PUT /users/42 with a full body sets the row to that document, and twice is the same as once. DELETE /sessions/abc converges — the second call may answer 404, but the state is identical, and idempotency is a claim about state, not about the response. "Set balance to 4,000" is idempotent; a conditional write guarded by a version number is idempotent by exclusion, since the second attempt fails the version check (see locking and concurrency control).

The unsafe ones all fold a read into the write. POST /charge, "add 100 to balance", INCR on a counter, appending to a list, sending an email. Their result depends on the state they started from, so a replay starts from a different state and lands somewhere new.

The HTTP spec labels GET, PUT and DELETE idempotent and POST not (see REST), but that is a contract you promise, not one the protocol enforces: a PUT handler that increments a counter inside is still broken. The only test that counts: run it twice with identical input, then compare final state.

The idempotency key

The client generates a UUID per logical operation — one key for "pay this cart", reused by every retry of it, never regenerated per HTTP attempt — and sends it as Idempotency-Key. The server stores key and result, and a repeat of the key returns the stored result rather than re-executing.

CREATE TABLE idempotency_keys (
  key           uuid PRIMARY KEY,      -- the unique index does the work
  request_hash  bytea NOT NULL,
  charge_id     uuid,
  response_body jsonb,
  created_at    timestamptz DEFAULT now()
);

BEGIN;
  INSERT INTO idempotency_keys (key, request_hash, charge_id, response_body)
  VALUES ($1, $2, $3, $4);          -- duplicate key raises a unique violation

  INSERT INTO charges (id, account_id, amount_paise, state)
  VALUES ($3, $5, 400000, 'captured');
COMMIT;

On a unique violation the handler rolls back, reads the stored row, and returns response_body with the original charge id. Nothing is charged twice.

The key row and the effect must commit in the same transaction. Both alternatives have holes. Write the charge first and the key second, and a crash in the gap leaves a committed charge with no key, so the retry charges again. The gap is small, which is why it is dangerous: at 50 QPS peak with a 5 ms gap, Little's Law gives 50 × 0.005 = 0.25 requests inside the window at any instant, so one unlucky deploy drops one and you hear about it from a customer, not a metric. Write the key first in its own transaction and the reverse breaks: if the charge then fails, the key is burned and every retry returns success for a payment that never happened. A lost write is worse than a duplicate, because nobody complains about it.

Store the request hash too and answer 422 when a key returns with different parameters, or a client bug becomes a way to read somebody else's result. Keys need only a short retention: 7 days of 1,000,000 requests/day is 7,000,000 rows at roughly 100 bytes, about 700 MB, and the lookup is a primary-key hit. One commodity Postgres box handles ~5,000 simple QPS, so 50 QPS of key checks is not the constraint.

Read the key from the leader, not a follower. With replica lag of even 200 ms, a retry arriving 50 ms later finds no key and re-executes — the same read-your-writes failure covered in consistency models.

A retried charge with and without an idempotency key100%−+FitSVGEsc

Scroll to zoom · drag to pan · 0 fits · Esc closes

When the effect lives in another system — a card network, an email provider — you cannot share a transaction with it. Commit the key as in_progress first, pass your own key to the external API as its idempotency key, and record the outcome on return. The outbox pattern is the same move applied to publishing: the event row is written inside the charge transaction, and a relay ships it afterwards.

Consumers dedupe on event id

For queues the key already exists: the producer stamps a stable event_id when the event is created, not when it is published. The consumer inserts that id into a processed_events table in the same transaction as its effect, and treats a unique violation as "already done, ack and drop". The dedupe table has to live in the database holding the effect; split them and the hole is back.

Say the trade-off out loud. Exactly-once delivery does not exist across a network — the ambiguity above is not an implementation gap. Exactly-once effect does, and idempotency is how you get it. Broker features sold as exactly-once are transactional writes plus dedupe inside the broker's own boundary; they say nothing about your database or the card network. See message queues and message brokers.

In an interview

What is being tested: whether you treat retries as a property of the system rather than an error path, and whether you reach for a transaction when two writes must agree.

What to say, when you draw a queue or a payment call: "delivery is at-least-once, so consumers dedupe on event id and the write endpoint takes an idempotency key — and the key row commits in the same transaction as the ledger entry, so a crash between them is impossible."

The mistake that loses points is answering "we'll use exactly-once delivery" — it says you have not thought about lost responses. Close behind: proposing the key but parking it in Redis outside the database transaction, which dies to "what if the process crashes between the charge and the Redis write". Third: deduping on user, amount and a five-minute window, which blocks a customer legitimately buying a second coffee.

Check yourself

An orders API takes 1,000,000 requests/day, 0.2% end ambiguously and each is retried once. How many duplicate orders per day, and does adding a second app server help?

1,000,000 × 0.002 = 2,000 ambiguous outcomes a day, roughly 1.4 per minute. The ones whose write had already committed replay in full, so budget close to 2,000 duplicate orders. More servers make it worse, not better: a retry can land on a different instance, so any in-memory dedupe cache misses. This is a protocol property, not a capacity one.

Your handler commits the charge to Postgres, then writes the idempotency key to Redis. Peak is 50 QPS and the gap is 5 ms. How exposed are you, and what changes?

Little's Law: 50 × 0.005 = 0.25 requests are inside the gap at any instant, so any crash, deploy or OOM kill in that window leaves a committed charge with no key and the retry double-charges. The loss is silent and unbounded per incident. Move the key into the same Postgres transaction behind a unique index; keep Redis in front only as a cache for the read path.

Which of these need an idempotency key: PUT /users/42 with a full body, POST /users/42/points with {"add": 10}, DELETE /sessions/abc?

Only the POST, because it folds a read into the write. PUT sets the resource to a value and DELETE converges. But idempotent is not safe under concurrency: two clients PUT different bodies, both replay-safe, and the last writer still wins — which is what version checks are for.