Webhook failures and retries: keep integrations from losing or duplicating work

Separate delivery from processing

Trace one event across the provider's delivery attempt, your endpoint, any queue and the downstream application. Record a correlation identifier at each stage. A successful HTTP response means only what your implementation makes it mean; it does not necessarily prove that the customer record was updated.

Verify signatures using the provider's documented method and the correct raw request data. Reject invalid deliveries, avoid logging secrets and keep endpoint configuration separate from application credentials. Check whether a proxy or body parser changes the signed payload before verification.

Acknowledge only after durable acceptance

Keep the receiving endpoint focused on validation and recording the work. When processing takes time, place accepted events in a durable queue and return the documented successful response after acceptance. A long synchronous operation can exceed the provider's timeout even when the downstream change eventually succeeds.

Stripe's webhook guidance documents retries, duplicate events and the absence of guaranteed event ordering. Those behaviours are useful reminders, but every provider has its own contract. Confirm its timeout, retry period, event identifiers and response expectations rather than borrowing another service's assumptions.

• Store the provider event identifier and processing state.
• Keep failed jobs available for investigation and controlled replay.
• Monitor queue age as well as queue length.

Make repeated delivery safe

Use a durable processing record to recognise an event already completed or accepted. Protect downstream side effects with a stable business-operation key where the destination supports it. A duplicate event should not send another email, create another invoice or apply another credit.

Stripe's idempotent-request documentation illustrates one provider's mechanism for recognising repeated operations. Understand the scope and retention of your chosen mechanism. An in-memory flag lost during restart is not a sufficient record, and using a new key for each retry defeats the purpose.

Handle stale events and recover deliberately

If a provider does not promise ordering, use its documented version or current-state retrieval approach before applying a potentially stale change. For an order-status update, a delayed 'created' event should not overwrite a later cancellation. Define what to do when the source cannot be reached for confirmation.

After an outage, compare source and destination records for the affected interval. Repair confirmed gaps through a bounded replay or reconciliation process. Distinguish temporary failures from invalid requests and permissions problems; repeated retries will not fix a missing required field or revoked access.