Journal
Work
7 Jun 2023#aws#serverless#ops 1 min read

The DLQ handler's first real failure

A dead-letter handler is only tested when things fail.


The dead-letter-queue handler had been in place for six months and had never seen a real failure. The first poison message needed a "major fix" twice in one day.

The handler assumed it would see a message once; it saw it three times. It assumed the payload would look like the happy path; it did not. Two changes stuck: the handler became idempotent, and the retry policy became explicit instead of whatever the queue defaulted to.

Takeaway

Test the failure path with a real failure before you need it.

Client engagement, 2021 to 2026. Names, identifiers and internals are generalised.