Connectors · Reliability

Which failure is safe to replay

Sort five groups of failed calls after an outage into replay as is, replay after a check, and do not replay, and name the check for each.

casemediumExercise · 3 min · 38 of 75

On this page

After an outage the dead-letter queue is a list of decisions. "Replay all" is one of them, and on this queue it double-sends reminders and overwrites three contacts staff fixed this morning.

The situation

The host that runs the agency's connectors could not reach other systems from 06:00 to 14:00 on Saturday. Every outbound call timed out, so whether each request arrived is unknown. On Monday at 08:00, about 50 hours later, the dead-letter queue holds five groups:

Group	Count	What failed
A	1,140	`GET /contacts` lookups against the CRM
B	31	`POST /invoices` to the accounting system, which accepts no idempotency key; each invoice carries its timesheet id in a reference field
C	26	`PUT /contacts/:id` full-record updates to the accounting system, built from the CRM on Saturday; staff edited 3 of those contacts there at 07:30 today
D	212	`payment.received` events stored by the receiver, which runs elsewhere, whose processing failed; the worker dedupes on event id, keys expiring after 72 hours
E	9	payment-reminder emails, sent with an `Idempotency-Key` the email provider honours for 24 hours

Voiding a duplicate invoice takes 15 minutes at $30 an hour; a client reminded twice costs a 10-minute founder apology at $100 an hour. The lesson is idempotency and deduplication.

The question

Decide, for each group, whether to replay it as it is, replay it after a check, or not replay it, and name the check.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

The test is not "is this call idempotent" but "is repeating it now, 50 hours later, harmless".

A: replay as is, or drop. Reads have no effect to repeat. Better still, drop them and let the jobs that needed them run again, since a Saturday lookup is stale anyway.

D: replay as is. The events are 50 hours old and the keys last 72, so any event already applied before the failure is recognised and dropped. Marking an invoice paid sets a state, which is safe twice.

B: replay after a check. A timeout means some of the 31 invoices may exist. Search by reference first, 31 requests, about a minute at 30 a minute, and create only the missing ones. Blind replay risks up to 31 duplicates at $7.50 each, plus any that reach clients.

E: replay after a check. The provider's key expired at 24 hours, so it will not catch a repeat. Check its sent log for the nine; resend only those with no record whose invoice is still unpaid once D is applied. A blind replay risks nine $17 apologies for 30 seconds of saved checking.

C: do not replay. A PUT is idempotent and still unsafe here. Replaying Saturday's full records would overwrite the three contacts staff fixed this morning with older data. Rebuild the 26 updates from the CRM's current state, sent as conditional writes on the record's version, so that a record changed since is refused and goes to a person.

Evidence that would change this: if the accounting system honoured an idempotency key with a window over 50 hours, B could be replayed as is.

Rubric

applies a test of "harmless to repeat now", not "idempotent by HTTP method"
replays D as is because the 72-hour expiry outlives the 50-hour gap, and states that comparison
checks B by reference before creating, with the request cost shown
notices that E's 24-hour key has expired and checks the provider's sent log
refuses to replay C because stale full-record writes overwrite newer edits, and offers a version-checked rebuild
names the evidence that would let one checked group be replayed as is

Where answers go wrong

"Replay everything; the connector is idempotent." It is idempotent within its key windows, and two of the five groups are outside them: E's key expired a day ago, and C's records are no longer the latest. The reviewer is checking whether the answer compares each window with the 50 hours.

"Replay A, B, D and E, and skip C because PUT is the risky one." The answer got C right for the wrong reason, then let B and E through. Of the writes here, a PUT is the only one HTTP calls idempotent; the risk in C is staleness, and the risk in B and E is a missing key.

"Delete the queue and run a full sync." A full sync rebuilds reads, but it does not send the 9 reminders, it cannot tell which of the 31 invoices exist, and it overwrites C's three edits the same way a replay would.