2026-09-18, Engineering · Tutorials

Webhook receiver design: retries, duplicates and ordering

A reliable webhook receiver never loses a result and never processes one twice, whatever happens to your endpoint, your database or the network. Sender retries are bounded: at most 10 attempts over roughly 21 to 43 minutes. The pattern below handles crashes, deploys, duplicates and out-of-order arrivals, and a small reconciliation job covers outages longer than the retry window.

It assumes you already verify webhook signatures and covers everything after verification.

The delivery contract you are designing for

Every design decision below follows from these documented properties of deliveries (see webhooks):

The retry schedule

Attempt 1 happens when the task finishes. Before each later attempt the service waits a base delay that starts at 5 seconds and doubles, and each wait is shortened by a random amount of up to half. The cumulative figures below ignore the time each attempt itself takes (up to 15 seconds).

A restart or deploy that takes a minute or two is covered by retries. An outage of more than about 21 minutes can exhaust a delivery’s attempts, so plan for that case explicitly.

The durable pattern: verify, write, acknowledge, process

The inbox row is written before the 2xx goes out, so there is no moment at which a delivery has been acknowledged but not stored.

Split the receiver into two parts with a durable store between them.

1. The HTTP handler verifies the signature, drops test deliveries, writes the raw delivery to an inbox table keyed by task.id, and returns 2xx. Nothing else. It should answer in milliseconds, far inside the 15-second limit.
2. A worker reads unprocessed inbox rows, does the real work (parsing results, updating dashboards, calling other services), and marks each row processed.

Why this shape:

• The write happens before the acknowledgment. If the process dies after the 2xx, the result is already stored. If it dies before, no 2xx went out and the delivery is retried. There is no window in which a result is acknowledged but lost.
• The primary key does the deduplication. A duplicate delivery hits the same task_id and inserts nothing.
• Slow or failing downstream work cannot cause retries. A bug in result processing leaves rows unprocessed in your inbox; it does not turn into timeouts, retry storms and dead deliveries.
• The handler has few dependencies. It needs the signing secret and one table, so it can stay up while the rest of the application is being deployed or migrated.

The inbox table (Postgres)

create table webhook_inbox (
  task_id      text primary key,
  body         jsonb not null,
  received_at  timestamptz not null default now(),
  processed_at timestamptz
);

create index webhook_inbox_unprocessed on webhook_inbox (received_at) where processed_at is null;

The handler (Flask and psycopg)

verify_signature is the stdlib function from the signature verification post.

import json
import os

from flask import Flask, request
from psycopg_pool import ConnectionPool

from verify import verify_signature

app = Flask(__name__)
pool = ConnectionPool(os.environ["DATABASE_URL"])
SECRET = os.environ["WEBHOOK_SECRET"]


@app.post("/hooks/answers")
def answers():
    raw = request.get_data()
    if not verify_signature(raw, request.headers.get("Webhook-Signature"), SECRET):
        return "invalid signature", 400
    delivery = json.loads(raw)
    if delivery.get("test"):
        return "", 204  # verified sample from the dashboard: acknowledge, never store
    with pool.connection() as conn:  # commits when the block exits
        conn.execute(
            "insert into webhook_inbox (task_id, body) values (%s, %s::jsonb) on conflict (task_id) do nothing",
            (delivery["task"]["id"], raw.decode()),
        )
    return "", 204

A duplicate returns 204 like the first delivery; answering it with an error would trigger more retries of something you already have.

The worker

from psycopg.types.json import Jsonb


def process_batch(limit: int = 100) -> int:
    with pool.connection() as conn:
        rows = conn.execute(
            """select task_id, body from webhook_inbox
               where processed_at is null
               order by received_at
               limit %s
               for update skip locked""",
            (limit,),
        ).fetchall()
        for task_id, delivery in rows:
            task = delivery["task"]
            conn.execute(
                """insert into answers (task_id, job_key, status, created_at, response)
                   values (%s, %s, %s, %s, %s)
                   on conflict (task_id) do nothing""",
                (task_id, task["idempotencyKey"], task["status"], task["createdAt"], Jsonb(delivery["response"])),
            )
            conn.execute("update webhook_inbox set processed_at = now() where task_id = %s", (task_id,))
    return len(rows)

for update skip locked lets several workers run in parallel without claiming the same rows. Each batch commits as one transaction, so a crash mid-batch leaves every row in it unprocessed for the next run. If one malformed row can make processing raise, record the error on that row and move on, or a single bad row will block its whole batch forever.

For a COMPLETED task, response.result holds the engine’s structured answer. For a FAILED task, response holds the error; store it as-is and keep it out of your metrics.

Deduplication: use task.id, not Webhook-Id

Every delivery carries a Webhook-Id header, stable across retries and redeliveries. It is tempting as a deduplication key, but it is not covered by the signature. task.id is inside the signed body, identifies the unit of work, and is what you poll with, so it is the key for the inbox, the results table and reconciliation.

Deduplication also neutralizes replays. A captured request replayed within the signature’s 300-second tolerance still verifies; with the inbox keyed by task.id it inserts nothing.

Out-of-order results

Deliveries for different tasks are sent concurrently and retried independently. A task that finished first can be delivered last if its first attempt failed. Three rules keep that from corrupting your data:

1. Never use arrival time as event time. Use task.createdAt for when you asked, and your own run key for which run a result belongs to.
2. Carry your job identity in the task. The body includes task.idempotencyKey. If you derive keys from meaning, as idempotency keys explained recommends (prompt, market, day), the delivery tells the worker which logical job it answers without a lookup. A task created without a key has a null idempotencyKey, so always send one.
3. Resolve conflicts deterministically. If one logical job ends up with more than one task, for example because you resubmitted after a failure under a new key, pick the winner by rule rather than by whichever arrived last. A reasonable rule: the most recently created COMPLETED task wins, and a FAILED task never replaces a COMPLETED one.

If a resubmitted task’s key is the original key plus a suffix (-r1, -r2), store the base key as job_key. The reporting query then reads:

select distinct on (job_key) *
from answers
where status = 'COMPLETED'
order by job_key, created_at desc;

Redelivery and reconciliation

A delivery that runs out of attempts, or targets a URL that isn’t https or doesn’t resolve to a public address, is marked dead. You have two recovery paths.

Redelivery from the dashboard

The Webhooks page lists deliveries, filterable by pending, delivered and dead, with attempts, the last status your endpoint answered, the last error, and timing. Owners can redeliver a delivered or dead delivery: it is sent again at once with a fresh set of attempts, keeps its Webhook-Id, and is signed with the secrets valid when it is sent. A pending delivery cannot be redelivered (it is still retrying on its own), nor can one whose task is no longer kept. Redeliveries are limited per account per hour, so this suits a handful of failures.

Reconciliation through the API

For a longer outage, poll the tasks you know about. GET /v1/async/task/{taskId} returns task, credits and, once the task has finished, response, the same body a delivery carries. Feed final ones into the same inbox and the worker handles them identically.

This requires recording each task id when you create the task:

import os
import time

import requests

API = "https://api.answerline.dev"
HEADERS = {"Authorization": f"Bearer {os.environ['ANSWERLINE_API_KEY']}"}


def reconcile(limit: int = 500) -> None:
    with pool.connection() as conn:
        rows = conn.execute(
            """select s.task_id from submitted_tasks s
               left join webhook_inbox i on i.task_id = s.task_id
               where i.task_id is null and s.created_at < now() - interval '1 hour'
               order by s.created_at
               limit %s""",
            (limit,),
        ).fetchall()
    for (task_id,) in rows:
        res = requests.get(f"{API}/v1/async/task/{task_id}", headers=HEADERS, timeout=30)
        if res.status_code == 429:
            time.sleep(1)  # not admitted; pick the rest up next run
            continue
        if res.status_code == 404:
            continue  # log it: the task is unknown or no longer kept
        res.raise_for_status()
        if res.json()["task"]["status"] not in ("COMPLETED", "FAILED"):
            continue  # still queued or running; a later run or a delivery will catch it
        with pool.connection() as conn:
            conn.execute(
                "insert into webhook_inbox (task_id, body) values (%s, %s::jsonb) on conflict (task_id) do nothing",
                (task_id, res.text),
            )

The one-hour threshold is illustrative. Choose it from the lifecycle: a task has 4 hours from creation to start (after that it fails uncharged with QUEUE_WAIT_EXCEEDED) and five minutes from its first start to finish, and its delivery retries run for up to about 43 minutes after that. A task with no stored result well past that horizon needs investigation, not more waiting. Treat only COMPLETED and FAILED as final; a task can go from PROCESSING back to QUEUED.

The job pauses on 429 and runs in bounded batches. GET /v1/async/status returns queuedTasks and processingTasks, so after an outage you can confirm no tasks are still pending before declaring reconciliation done.

A note on task creation: if a create request times out and your retry with the same idempotencyKey gets 409 RESOURCE_CONFLICT, the task exists but you may not have its id. The delivery body carries task.idempotencyKey, so match on the key when it arrives and fill in the id.

Planned maintenance

For scheduled downtime, you have better options than relying on retries:

• Keep the handler separate. If the webhook handler is a small service that only verifies and writes to the inbox, application deploys and migrations do not affect it.
• Stop submitting before the window. Tasks you have not created cannot produce deliveries. DELETE /v1/async/queue removes tasks still QUEUED (never charged) if you need to stop work already submitted; running and finished tasks are unaffected.
• Reconcile afterwards. Run the reconciliation job once the receiver is back, instead of redelivering one by one.

Choosing status codes

Because every non-2xx is retried, status codes are instructions to the sender:

A spike in signature failures has two usual causes: a rotation deployed out of order, or someone probing the URL. The delivery log distinguishes them. If the dashboard shows your endpoint answering 400 to real deliveries, it is your configuration.

Failure modes to test

Run each of these against staging before relying on the receiver:

• Kill the process between the insert and the response. The delivery is retried, the insert is a no-op, the worker processes the row once.
• Send the same signed body twice concurrently. Expect exactly one inbox row.
• Make the worker raise on one row. Other rows still get processed; the handler still answers 2xx.
• Stop the database for a minute. The handler answers 5xx, deliveries retry, nothing is lost once the database is back.
• Deliver two results for one job_key in reverse order. The reporting query still picks the correct one.
• Send a dashboard test delivery. It verifies, gets a 2xx and creates no row.
• Put the receiver behind a redirect (for example http to https, or a missing trailing slash). Confirm attempts fail, then register the final URL.
• Take the receiver down for longer than the retry window. Confirm dead deliveries show in the log and the reconciliation job recovers every result.

Checklist

• Handler: verify, skip "test": true, insert into inbox by task.id, 2xx
• Handler answers well within 15 seconds and depends only on the secret and the inbox
• Worker processes inbox rows idempotently, in parallel-safe batches
• Results keyed by task.id; job identity from task.idempotencyKey
• Conflict rule that never lets a FAILED replace a COMPLETED, independent of arrival order
• Task ids recorded at creation; reconciliation job with a horizon derived from the task lifecycle
• Alert on signature failures and on dead deliveries

For the trade-offs between webhooks, polling and synchronous calls, read sync calls, async tasks or webhooks, and for handling API errors on the submitting side, handle API errors like a pipeline. The task lifecycle is in async tasks and the delivery details are in webhooks.

Keep reading