Lab Objective

Practice the reasoning behind a safe backup verification workflow: restore to an isolated location, inspect the restored state, and avoid starting a second live client with copied credentials.

Lab Environment

  • Backup concept: encrypted off-host repository
  • Database type: SQLite in WAL mode
  • Restore target: isolated recovery directory or separate machine
  • Safety rule: do not start the live service from restored credentials

Scenario

An automation service has a live SQLite database, configuration, and authentication material. A raw copy of a WAL-mode database can be inconsistent while the service is writing. A backup is only useful if it can be restored and inspected somewhere separate from the live process.

Step 1 - Create a Consistent Database Snapshot

Use SQLite’s own snapshot capability instead of copying a live database file directly:

VACUUM INTO 'state-snapshot.db';

This produces a consistent snapshot at the SQLite level. It avoids treating a changing WAL-backed file as a stable artifact.

Step 2 - Restore to an Isolated Location

mkdir -p /tmp/recovery-lab
# restore-tool restore latest --target /tmp/recovery-lab
find /tmp/recovery-lab -type f | wc -l

In a real recovery plan, the target should be a different host or a clearly isolated directory. Do not overwrite the live service’s state while testing.

Step 3 - Inspect Database Integrity

sqlite3 /tmp/recovery-lab/state-snapshot.db 'PRAGMA integrity_check;'
sqlite3 /tmp/recovery-lab/state-snapshot.db '.tables'

Do not stop at “the archive listed files.” Read the restored database and check the tables expected by the application.

Step 4 - Separate Derived-Index Warnings from Corruption

Search-index structures can be rebuilt. Core tables, credentials, and configuration cannot be treated the same way.

Finding Response
Core table integrity error Fail recovery verification
Missing expected configuration Fail recovery verification
Derived search-index compatibility warning Investigate and document; do not automatically call the entire backup corrupt

Step 5 - Verify Non-Interference

Before and after the restore check, record only safe metadata from the live system:

pgrep -f 'gateway|bridge'
shasum -a 256 /path/to/live/credential-file

The restore verifier should never launch a second copy of a messaging client or use restored credentials to connect to a live service.

Security Takeaways

  • Encryption protects backup confidentiality, but does not prove recoverability.
  • WAL-mode databases need consistent snapshots.
  • A restore test must happen outside the live service path.
  • Read restored data structurally; archive existence is weak evidence.
  • Treat derived-index warnings differently from core-state corruption.

Reflection

This lab showed why recovery is an operational security skill. The question is not “did the backup job run?” but “can I restore safe, readable state without making the incident worse?”

TL;DR

Create a consistent snapshot, restore it elsewhere, read it, and prove the verification did not touch the live service.