Work
7 Jun 2023#aws#serverless#ops 1 min read
The DLQ handler's first real failure
A dead-letter handler is only tested when things fail.
The dead-letter-queue handler had been in place for six months and had never seen a real failure. The first poison message needed a "major fix" twice in one day.
The handler assumed it would see a message once; it saw it three times. It assumed the payload would look like the happy path; it did not. Two changes stuck: the handler became idempotent, and the retry policy became explicit instead of whatever the queue defaulted to.
Takeaway
Test the failure path with a real failure before you need it.
Client engagement, 2021 to 2026. Names, identifiers and internals are generalised.