A Backup You Have Never Restored Is a Hope

A Backup You Have Never Restored Is a Hope

Ask a small business whether it has backups and the answer is nearly always yes. Ask when someone last restored one onto a clean machine and checked what came back, and the answer is usually a pause. Until that pause turns into a date, what you have is not a backup. It is a hope with a storage bill.

Backups fail in boring ways. The job stopped running months ago and nobody was told. The dump is there but will not load. The database came back fine and the uploaded files did not, because nobody was copying them. None of it shows until the day you need the backup.

Three copies, two kinds, one elsewhere

The old 3-2-1 rule still holds. Keep three copies of your data, on two different kinds of storage, with one copy somewhere else entirely. The live database counts as one. A nightly dump on the same server counts as a second, but only just, because it dies with the server. The third has to live with a different provider, or at least under a different account, so that one bad day (a stolen hosting login, a billing dispute, a mistyped delete) cannot reach all three.

If the server can delete every backup, so can anyone who breaks into the server.

Hosting providers often take backups for you. That is useful, and those backups are theirs: you restore on their schedule, through their panel, while your account is in good standing. Shared hosting, which this site runs on, is a fine home for a website. It is not a backup plan on its own.

The database is not the whole system

Picture restoring onto a fresh server tonight, with the old one gone. The database dump gets you the rows. You would also need:

  • Uploaded files. Invoices, photos, signed documents. These usually sit on disk, and the database only holds their names. Restore the database without them and every link points at nothing.
  • Configuration. The settings file, the web server configuration, the scheduled jobs, the version of the language and its extensions. Small, easy to forget, slow to rebuild from memory.
  • The secrets you would need. Database passwords, API keys, and above all the key to anything stored encrypted, the backups included. Keep them apart from the data, so a stolen backup does not arrive with its own keys, but somewhere you can reach when the server cannot. An encrypted backup whose key lived only on the dead server is a very secure way of having nothing.
  • The steps. A restore you have done once becomes a page of notes. Keep that page next to the backups.

Restore on a calendar

Restore tests are dull, which is why they get skipped, which is why they belong on a schedule rather than on good intentions. Monthly suits most small systems, plus one straight away after any change to how backups are taken. Restore somewhere separate, never over the live system, and check that the data makes sense, not merely that the command finished.

The smallest useful version, for MySQL, loads last night's dump into a scratch database and checks two things: that the important table has rows, and that the newest one is recent.

# restore-test.sh: run monthly from cron; a non-zero exit should alert someone
set -euo pipefail

DUMP=/backups/nightly/latest.sql.gz

mysql -e "DROP DATABASE IF EXISTS restore_test; CREATE DATABASE restore_test;"
gunzip -c "$DUMP" | mysql restore_test

ROWS=$(mysql -N -e "SELECT COUNT(*) FROM restore_test.orders;")
AGE=$(mysql -N -e "SELECT DATEDIFF(NOW(), MAX(created_at)) FROM restore_test.orders;")

if [ "$ROWS" -eq 0 ] || [ "$AGE" -gt 1 ]; then
  echo "RESTORE TEST FAILED: rows=$ROWS, newest row $AGE days old"
  exit 1
fi
echo "Restore test passed: rows=$ROWS, newest row $AGE days old"

Notice what the script does not contain: a password. The mysql client reads it from its own option file, which is where it belongs.

It catches the two failures that matter most: a dump that will not load, and a dump that loads perfectly but is weeks old because the job that makes it stopped quietly. Make a failure reach a person. A restore test that fails silently is just another hope.

Once or twice a year, go further. Rebuild the whole system from backups on a clean machine, files, configuration and secrets included, and time it. That number is how long you would be down on the real day. Far better to learn it on a quiet Tuesday.

Nobody wants a backup

Nobody has ever wanted a backup. What people want is a restore: on the bad day, quickly, with nothing missing. The backup is the part you can buy in advance. The restore is the part you have to practise.