Queue test cases, for the job that runs twice

Every mainstream queue is at-least-once, which means duplicate delivery is a normal operating condition rather than a fault. The handler is therefore the place where exactly-once has to be constructed, and it is constructed by making the side effect idempotent. A handler that charges a card, sends an email or posts to a ledger without a deduplication key will do it twice in production, and the trigger is usually not an exotic failure: it is simply a job that took longer than its visibility timeout, so the broker concluded the worker had died and handed the message to somebody else while the original was still running.

Poison messages are the failure that stalls a queue while every dashboard says it is healthy. A message that fails deterministically will be retried forever without a dead letter queue, consuming a worker and blocking whatever is behind it. The dead letter queue is only half the fix: it also needs to be inspectable and replayable, because otherwise the failed messages accumulate somewhere nobody looks and the events they represent are quietly lost.

Scheduling breaks the moment you scale out. Three application instances each holding the same cron definition will run every scheduled job three times unless a lock enforces single execution, and this is invisible in a single instance environment. The neighbouring cases are overlap, where a run that takes longer than its interval starts concurrently with itself, and timezone, where a daily job across a daylight saving transition either runs twice or not at all.

Finally, two ordering problems that look unrelated and are the same mistake. Enqueuing inside a transaction that later rolls back sends a worker after a record that does not exist. And deploying while jobs are in flight kills them midway unless the shutdown grace period returns them to the queue cleanly. Both come from treating the queue as though it were inside your transaction boundary when it is emphatically outside it.