Connectors · Building your own

A connector for a system with no webhooks

Design a polling connector for a tool with no webhooks: interval, cursor, overlap, deletes and alarms, with the interval defended in requests.

designhardExercise · 3 min · 40 of 75

On this page

Without webhooks, every property a connector needs has to be built from one list endpoint and a clock. The design either fits inside a shared limit of 20 requests a minute, or it takes the time-tracker sync down with it.

The situation

The agency's project-management tool has a REST API and no webhooks. It holds 6,000 tasks, 1,500 of them on active projects, and about 400 change on a working day. The list endpoint returns 100 tasks a page, filters by updated_since to the second, and sorts by updated_at. Each task carries an integer version. A deleted task simply stops appearing. The limit is 20 requests a minute per account, and the time-tracker sync already uses up to 8.

The automation: when a task moves to "ready for client review", the model drafts a short update for the client from the task's notes, and an account manager approves it before it is sent. The account managers want each draft within 10 minutes. A missed task is a client update nobody sends. A duplicated one is two drafts in the queue and, if both are approved, two emails to the client. The lesson is webhooks versus polling.

The question

Design the connector: what it polls and how often, the cursor it keeps, and how it handles duplicates, deletes, the limit and alarms. Defend the interval with the arithmetic.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Poll the changed list every 2 minutes, from the cursor minus 60 seconds: 720 polls a day. Over a 9-hour working day, 400 changes is about 1.5 a poll, so nearly every poll is one page. That is about 720 requests a day, 0.5 a minute against the 12 the time tracker leaves, and a change is seen within 2 minutes, leaving 8 of the 10 for the model and the queue.

The cursor is the largest updated_at among the tasks committed from the last poll, saved in the same transaction. The overlap covers two changes in one second and a few seconds of clock skew. It also re-delivers tasks, so every write is an upsert by task id, applied only if version is newer than the stored one.

The draft is keyed on task id plus the version at which the task became ready. Seen twice at that version, it drafts once; sent back and made ready again, it correctly gets a new draft.

Deletes: each night, sweep the active tasks, 1,500 ÷ 100 = 15 requests, and compare ids. A missing task is marked deleted and its pending draft withdrawn. A weekly sweep of all 6,000, 60 requests, covers archived projects.

A limiter holds the connector to 10 a minute and honours Retry-After; a 4xx goes to the dead-letter queue. Alarms fire on no successful poll for 10 minutes, a cursor 10 minutes behind, or any dead letter at 09:00.

One finding would change this: a bulk status change that leaves updated_at alone. Test it once; if so, poll the list filtered by status instead.

Rubric

polls changed records with an overlap, and derives the interval from the 10-minute target and the spare limit with the arithmetic shown
keeps a cursor saved with the committed tasks, and says what the overlap protects against
makes writes idempotent on task id and version, and keys the draft so a re-delivered task drafts once
detects deletes with a sweep whose request cost is shown
paces under the shared limit, honours Retry-After, and alarms on a stalled poll or a lagging cursor
names the behaviour of the tool that would change the design, and how to test for it

Where answers go wrong

"Poll every 10 seconds to be safe." It costs 8,640 requests a day for a target the 2-minute poll already meets, and it still needs the overlap and the version check, so it buys nothing but load on a limit the time tracker shares.

"Read all 6,000 tasks every 10 minutes and diff." That is 60 requests a read, 8,640 a day, and 6 minutes of every 10 at the connector's share of the limit. It does find deletes, at the price of re-reading nearly 6,000 unchanged tasks every time.

"Set the cursor to the time the poll ran." A task saved while the poll was running carries an earlier updated_at than that cursor and is never seen again, and the agency's server clock is not the tool's. The reviewer is checking that the cursor comes from the data.