Backups that restore nothing

A backup job that exited 0 and restored zero rows, and the drill that caught it.

FromSelf-Hosted Platform: k3s on Hetzner

The nightly backup piped a full dump straight into a restore and exited 0 every night. The first real drill restored zero rows. Three causes stacked: a client-server version mismatch in the dump format, a schema shape pre-created by the chart bootstrap, and a parser error that cascaded silently.

The replacement uses a custom-format dump and a real restore step, and the drill now checks row counts for parts, users and storage objects against the source. The first cutover with it still had one surprise left.

flowchart LR
  D[pg_dump custom format] --> O[Object storage]
  O --> R[pg_restore into fresh release]
  R --> V{Row counts match?}
  V -->|yes| P[Pass]
  V -->|no| F[Fail loudly]

Takeaway

A backup that has never been restored is a hope, not a backup.