Connectors · Building your own

Backfill time under a rate limit

Time a 36,000-ticket backfill under a per-minute and a per-day limit shared with live traffic, and decide when to start it to meet a Monday go-live.

estimatemediumExercise · 3 min · 39 of 75

On this page

A backfill has two clocks: the per-minute limit everyone checks, and the per-day allowance most people find at 23:20. This one has to finish before a Monday go-live, on an account the live support automation also uses.

The situation

The agency is loading five years of support history from its old helpdesk into the new support-triage automation, so that triage can see each client's past problems. The helpdesk holds 36,000 tickets. Its list endpoint returns 100 tickets a page without their conversations; each ticket's conversation is one more call.

The helpdesk limits each account to 40 requests a minute and 20,000 requests a day, and the daily allowance resets at 00:00 UTC. The live support automation runs on the same account and uses up to 8 requests a minute and about 4,000 a day. Keep 2 a minute and 1,000 a day of headroom for retries.

It is Friday, 15:00 UTC. Triage goes live on Monday at 08:00 UTC, and the founder wants the full history loaded by then, with time to recover from one failed run. The lesson is backfills and building your own connector.

The question

Estimate when the backfill finishes if started now, and decide whether to start now or wait for the allowance to reset at midnight.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Requests first: 36,000 ÷ 100 = 360 list pages, plus 36,000 conversation calls, 36,360 in all.

The pace is 40 − 8 − 2 = 30 a minute. On that alone, 36,360 ÷ 30 = 1,212 minutes, 20.2 hours. But the daily allowance binds first. The backfill's share of a day is 20,000 − 4,000 − 1,000 = 15,000 requests, which at 30 a minute is used up in 500 minutes, 8 hours 20 minutes. After that the backfill waits for midnight.

Started now:

Friday     15:00 → 23:20    15,000 requests
Saturday   00:00 → 08:20    15,000 requests
Sunday     00:00 → 03:32     6,360 requests   (6,360 ÷ 30 = 212 min)

It finishes on Sunday at about 03:30 UTC, 36.5 hours after starting and 28.5 hours before go-live. That leaves room for one failed run, a fix, and a resume from the cursor.

Started at 00:00 on Saturday instead, it uses Saturday's and Sunday's allowances and 6,360 requests of Monday's, finishing at 03:32. That is 4.5 hours of slack, and one failure misses go-live. Start now, with the limiter at 30 a minute and a daily counter that stops the backfill at 15,000, so the live automation's share is never touched.

What would change this: a list parameter that embeds the conversation, which turns 36,360 requests into 360, 12 minutes at 30 a minute. Check the docs before anything else. Failing that, ask the provider for a temporary daily allowance for a migration; at 40,000 a day the per-minute pace binds instead, and the backfill finishes at about 11:10 on Saturday.

Rubric

counts list pages and detail calls separately and adds them, with the arithmetic shown
derives the backfill's per-minute pace from the limit, live traffic and headroom
finds that the daily allowance, not the per-minute limit, sets the finish time
lays out the run day by day across the midnight resets and gives a finish time
decides when to start with the slack before go-live stated, and caps the daily spend so live traffic keeps its share
names the evidence that would shorten the job, such as an embed parameter or a raised allowance

Where answers go wrong

"36,360 ÷ 30 is 20.2 hours, so it finishes Saturday at 11:12." It ignores the daily allowance. At 23:20 on Friday the backfill has used its 15,000, and if it keeps going it spends the live automation's share, and live support is refused until midnight. The reviewer is checking whether both limits were read.

"Use all 40 a minute: 909 minutes, about 15 hours." It takes the live automation's 8 a minute on a Friday afternoon, and it still hits the daily allowance at the same 15,000.

"Wait for midnight so the allowance is fresh." It sounds tidy and costs a day: Friday's 15,000 go unused, so it finishes at 03:32 on Monday with 4.5 hours to spare and no room for a failure.