Webhook test cases, for when the consumer stops answering

A webhook system makes two guarantees that most implementations do not state and most receivers do not expect: delivery is at-least-once, and order is not guaranteed. Both follow directly from retrying. If the sender does not attach a stable event identifier, the receiver cannot deduplicate, so every retry becomes a second order, a second charge or a second email. If the payload carries no sequence number or version, an updated event arriving before its created event leaves the receiver in a state that matches neither. Neither problem appears while everything is healthy, which is exactly why they ship.

The sender side failure is a dead consumer. One subscription whose endpoint has been unreachable for hours accumulates a backlog, and if that backlog shares a worker pool with every other subscriber, one broken integration degrades delivery for all of them. Bounding the backlog, timing out slow endpoints rather than waiting indefinitely, and disabling a persistently failing subscription with a notification through another channel are three separate cases and all three are load bearing.

Security on a webhook is unusual because the receiver has no session and no way to authenticate the sender beyond what the payload carries. That makes the signature the entire trust boundary: it has to cover the raw body plus a timestamp, so that altering the body fails verification and a captured request cannot be replayed a week later. Secret rotation needs an overlap window, otherwise rotating breaks every receiver at once. And the endpoint URL itself is untrusted input, so registering one that resolves to a private address turns your sender into a server side request forgery tool aimed at your own network.

Finally, two things that are easy to get subtly wrong. Payload scope: a webhook is an unauthenticated push, so it must never carry fields the subscriber could not have requested through the API. And emission timing: firing the event inside a transaction that later rolls back tells the receiver about a change that never happened, and there is no correction message that undoes what they already did with it.