Retries and Delivery Guarantees

Webhook consumers should assume that a delivery can arrive more than once and that downstream processing can fail after the sender receives an acknowledgement. Durable queues, idempotent handlers, and observable dead-letter flows are the safest operating model.

Receiver contract

1. Verify the signature using the raw request body.
2. Reject stale or duplicate delivery identifiers.
3. Persist the accepted event to a durable queue.
4. Return a 2xx response only after durable acceptance.
5. Process asynchronously and record success or failure.

HTTP status handling

Idempotency pattern

create table webhook_deliveries (
  delivery_id text primary key,
  event_type text not null,
  received_at timestamptz not null default now(),
  processed_at timestamptz
);

Insert the delivery id with a unique constraint before applying business changes. A duplicate insert means the same event has already been accepted and can be acknowledged without repeating the business action.

Processing deadlines

• Keep the acknowledgement path short and free of third-party API calls.
• Move CRM, storage, and analytics writes into the asynchronous worker.
• Apply bounded concurrency to prevent a retry burst from overwhelming downstream systems.
• Use exponential backoff with jitter for your own worker retries.
• Preserve the original event id, attempt count, and last error in observability tools.

Recovery workflow

1. 01

   Identify the failed cohort

   Filter delivery logs by event type, destination, time range, and response class.

2. 02

   Repair the receiver

   Resolve authentication, schema, capacity, or downstream dependency failures before replaying.

3. 03

   Replay from the immutable source

   Reprocess stored deliveries in bounded batches using the original event id.

4. 04

   Reconcile outcomes

   Compare business records, call identifiers, and CRM activity history for completeness.