Axel Verne → Junk Drawer kit → Chapter 11

Chapter 11 — Fortress II: Backups That Actually Restore

27 blocks to copy, in the order they appear in the chapter. Nothing here is required to be typed by hand.

Step 1: Install restic

11.1 · bash

curl -sL https://github.com/restic/restic/releases/download/v0.17.3/restic_0.17.3_linux_amd64.bz2 | bzip2 -d > /tmp/restic
sudo mv /tmp/restic /usr/local/bin/restic
sudo chmod +x /usr/local/bin/restic
restic version

Step 2: Prepare the USB drive

11.2 · bash

lsblk    # or `sudo fdisk -l` if lsblk doesn't show it

11.3 · bash

sudo mkfs.ext4 -L backup /dev/sdb1    # assuming one partition already exists
# Or, format the whole drive fresh:
# sudo parted /dev/sdb mklabel gpt
# sudo parted /dev/sdb mkpart primary ext4 1MiB 100%
# sudo mkfs.ext4 /dev/sdb1

11.4 · bash

sudo mkdir -p /mnt/backup-usb
sudo mount /dev/sdb1 /mnt/backup-usb
sudo chown $USER:$USER /mnt/backup-usb

Step 3: Initialize a restic repository on the USB drive

11.5 · bash

restic init -r /mnt/backup-usb

Step 4: Back up `/srv/lab/data` to USB

11.6 · bash

restic -r /mnt/backup-usb backup /srv/lab/data

11.7 · bash

restic -r /mnt/backup-usb snapshots

Step 5: Database-aware backups (critical)

11.8 · bash

# Immich (requires credentials from compose file)
docker compose -f /srv/lab/compose/immich/compose.yaml exec -T immich-server python3 -m immich dump_infra > /srv/lab/backup/immich-db.sql

# Vaultwarden (uses SQLite, just copy it)
cp /srv/lab/data/vaultwarden/db.sqlite3 /srv/lab/backup/vaultwarden-db.sqlite3

# Jellyfin (uses SQLite, same)
cp /srv/lab/data/jellyfin/jellyfin.db /srv/lab/backup/jellyfin-db.sqlite3

# Audiobookshelf (uses SQLite)
cp /srv/lab/data/audiobookshelf/database.sqlite3 /srv/lab/backup/abs-db.sqlite3

11.9 · bash

restic -r /mnt/backup-usb backup /srv/lab/backup

Step 6: Set up a second backup on your relative's box

11.10 · bash

restic init -r /srv/lab/backup/remote-repo

11.11 · bash

# Find the relative's Tailscale IP
tailscale list    # e.g., 100.x.x.x

# Create a restic repository pointing to their machine via SFTP
restic init -r 'sftp://user@100.x.x.x:/srv/lab/backup/remote-repo'

11.12 · bash

echo "my-password" > ~/.restic-sftp-pass
chmod 600 ~/.restic-sftp-pass

11.13 · bash

RESTIC_PASSWORD_FILE=~/.restic-sftp-pass \
restic -r 'sftp://user@100.x.x.x:/srv/lab/backup/remote-repo' \
backup /srv/lab/data

Step 7: Cloud backup to Backblaze B2

11.14 · bash

export RESTIC_REPOSITORY='s3:https://s3.us-west-002.backblazeb2.com/my-lab-backup'
export RESTIC_PASSWORD='your-restic-password'
export AWS_ACCESS_KEY_ID='your-b2-app-key'
export AWS_SECRET_ACCESS_KEY='your-b2-app-secret'

restic init

11.15 · bash

restic backup /srv/lab/data

Step 8: Automate with systemd

11.16 · ini

[Unit]
Description=Lab backup (USB, remote, cloud)
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
User=youruser
ExecStartPre=/bin/sleep 10
# USB backup
ExecStart=/bin/bash -c '
  set -e
  mount /dev/sdb1 /mnt/backup-usb || true
  cp /srv/lab/data/**/db* /srv/lab/backup/ 2>/dev/null || true
  restic -r /mnt/backup-usb backup /srv/lab/data /srv/lab/backup
  umount /mnt/backup-usb || true
'

[Install]
WantedBy=multi-user.target

11.17 · ini

[Unit]
Description=Lab backup timer (daily at 2 AM)

[Timer]
OnCalendar=daily
OnCalendar=*-*-* 02:00:00
Persistent=true

[Install]
WantedBy=timers.target

11.18 · bash

sudo systemctl daemon-reload
sudo systemctl enable --now lab-backup.timer
sudo systemctl status lab-backup.timer

11.19 · bash

sudo journalctl -u lab-backup.service -n 50

Make it good

11.20 · bash

restic -r /mnt/backup-usb forget --keep-daily 30 --keep-monthly 12
restic -r /mnt/backup-usb prune

When it breaks

11.21 · bash

echo 'UUID=your-uuid /mnt/backup-usb ext4 defaults,auto 0 2' | sudo tee -a /etc/fstab

11.22 · bash

sudo smartctl -A /dev/sdb

Drill 1: Restore a single file from USB

11.23 · bash

restic -r /mnt/backup-usb snapshots   # note the latest snapshot ID
restic -r /mnt/backup-usb ls <snapshot-id> | head    # list files

11.24 · bash

restic -r /mnt/backup-usb restore <snapshot-id> --target /tmp/test-restore
ls /tmp/test-restore/srv/lab/data/vaultwarden/

Drill 2: Restore Immich database from backup dump

11.25 · bash

restic -r /mnt/backup-usb restore <snapshot-id> --target /tmp/test-restore
cat /tmp/test-restore/srv/lab/backup/immich-db.sql | head -20

Drill 3: Full system restore (simulation)

11.26 · bash

RESTIC_PASSWORD_FILE=~/.restic-sftp-pass \
restic -r 'sftp://user@100.x.x.x:/srv/lab/backup/remote-repo' \
snapshots

11.27 · bash

export RESTIC_REPOSITORY='s3:https://s3.us-west-002.backblazeb2.com/my-lab-backup'
export RESTIC_PASSWORD='your-restic-password'
export AWS_ACCESS_KEY_ID='your-b2-app-key'
export AWS_SECRET_ACCESS_KEY='your-b2-app-secret'

restic snapshots
restic restore latest --target /tmp/lab-restore-cloud

← Chapter 10Chapter 12 →