Connectors · Reliability

The retry loop that retries a 400

Find the three defects one weekend's log exposes in a contractor's retry wrapper, cost the worst, and replace it with a policy sorted by status.

critiqueeasyExercise · 3 min · 37 of 75

On this page

Ten lines of wrapper decide how every call in the invoice run fails. One weekend's log shows them failing three different ways, and only one of the three would show up in a test.

The situation

The agency's invoice run calls the accounting system through a wrapper a contractor wrote:

def call(method, url, body=None):
    for attempt in range(6):
        try:
            r = http(method, url, json=body, timeout=10)
            if r.status < 300:
                return r
        except Timeout:
            pass
        sleep(2)
    raise Failed(url)

The run restarts every 15 minutes until every invoice has gone out. The API allows 30 requests a minute per account, shared with the overdue reminders. It accepts no idempotency key, but each invoice carries its timesheet id in a reference field. The weekend's log shows three things:

One client's invoice returned 422 tax_number missing six times per run, from 17:00 Friday to 09:00 Monday.
A POST /invoices for another client timed out at 10 seconds; the retry returned 201. INV-0533 and INV-0534 now exist for the same week.
In the 17:00 burst, nine calls got 429 with Retry-After: 20. Each retried five times, 2 seconds apart, and failed.

Assume staff time at $30 an hour. The lesson is rate limits, retries and backoff.

The question

Find the three defects the log exposes, put a number on the first, and say what replaces the wrapper.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

The wrapper treats every failure as transient, and the log shows three that are not.

The 422 is the request, not the server. The same body fails the same way, so each run spends six requests learning nothing. Friday 17:00 to Monday 09:00 is 64 hours, 256 runs, 1,536 refused requests on the limit the reminders share, and the invoice still has not gone out. Nobody was told, because the error lives only in a log. A 4xx should never retry: it goes to a dead-letter queue with the response body, and the operations manager hears within the hour and fixes the client's record in five minutes.

The timeout on a POST is ambiguous: the invoice was created and the reply lost, so the retry created INV-0534. After a timeout on a create, search by reference and resend only if nothing is found. Voiding the duplicate costs 15 minutes, $7.50, provided it is caught before a client sees it.

The 429s were told to wait 20 seconds and retried after 2. Five retries 2 seconds apart span 10 seconds, so all five landed inside the 20 and failed. Wait for the larger of Retry-After and the backoff, and pace the 17:00 burst with a client-side limiter at 30 a minute so the 429s never happen.

The replacement sorts by status: 2xx done; 429 waits; a 5xx or timeout on a read backs off 1, 2, 4, 8 seconds with jitter; a timed-out create looks up first; a 4xx dead-letters and alerts. One thing would make the create retryable as it stands: an API that honours an idempotency key.

Rubric

names the three defects separately: a 4xx retried, a write with no key retried after a timeout, Retry-After ignored
costs the 422 loop in requests over the weekend with the arithmetic shown, and notes that nobody was told
replaces the blind POST retry with a lookup by reference before any resend
treats Retry-After as a minimum wait and adds client-side pacing for the burst
proposes a policy sorted by status with a dead-letter queue and an alert, not a better-tuned sleep
names the evidence that would make the create safe to retry

Where answers go wrong

"Use exponential backoff instead of a fixed 2 seconds." That fixes one defect at most. A 422 retried with exponential backoff is still retried, and the POST still duplicates. The reviewer is checking whether the answer reads the status before deciding to retry.

"Cut the attempts from six to three." It halves the wasted requests and keeps every defect: the 422 still loops all weekend, and one retry is enough to create INV-0534.

"Catch every failure and email the founder." Every 429 and 503 becomes an email, the emails are filtered within a week, and the 422 is missed again. An alert belongs to a failure a person must fix, which here is only the 422.