niklas‑stephan.de

Backup setup: restic + Backrest farm, repos on the Pi 5 hestia (since 2026-10-21)

Last updated 05/11/2026 · backup, restic, backrest, hestia, iris, parasanoid

CURRENT 2026-10-21: repos on the Pi 5 hestia (replaces the Deco share)

  • Destination: restic rest-server on hestia (restic-server container, /data/docker/restic-server/, port 8000, auth off, append-only off) writing to the ext4 disk /srv/backup/restic/{metis,iris,handtrixxx}. Details of the box, disk and speeds: memory pi-backup-server.
  • Repo URIs in the Backrest configs: metis rest:http://192.168.68.54:8000/metis/, iris rest:http://192.168.68.54:8000/iris/, handtrixxx.com rest:http://100.64.0.4:8000/handtrixxx/ (tailnet, direct path). No CIFS mounts, no /mnt/deco, no iris relay any more (iris restic-server project removed, fstab lines removed, /mnt/deco deleted).
  • Repos were restic inited fresh on 2026-10-21 ~12:08 UTC (old NTFS repos wiped with the disk) and re-seeded from scratch. First runs triggered right after: metis-data (~65 MB/s), iris-data (done in seconds), handtrixxx data/sources/apps/media queued on handtrixxx.com. Media (3.1 TB) is limited by the home uplink (~19 MB/s measured to the Pi, so about 2 days); the daily cron plans (handtrixxx-media 00:00, moved from 20:00 on 2026-10-23 because prime time) retry automatically if a run is killed, restic skips already-uploaded data. No manual step is needed.
  • Why the Deco was dropped: the BE65-5G dropped its LAN link twice on 2026-10-21 (10:00:45 and 11:40:13, ~100 s each, a router-reboot signature seen simultaneously by metis and iris) plus 184 CIFS -32/-104 connection resets from its SMB1 server since 10-19; iris had no link drops in its two earlier boots. Backups as the trigger is likely but was not proven. Backrest compose backups before the change: compose.yaml.bak-pre-pi and config/config.json.bak-pre-pi-20261021 on each host.
  • Repoint helper (kept in the workspace): /data/maintenance/backrest-repoint.py <repo-id>=<new-uri> (run with sudo, container stopped; inits the repo with the stored password and rewrites config.json, never prints secrets). Backrest repo guid is a hash of the repo id, so it stays valid after changing the URI.
  • Not append-only on purpose (Backrest prunes), auth off on purpose (LAN and tailnet only), both Niklas’s decisions.

Setup 2026-10-19..21 (HISTORICAL: Deco SMB1; the plans, schedules, sizes and compose gotchas below still apply, only the destination changed)

  • Parasanoid is dead as the backup tool (archived). The live logic: Backrest (garethgeorge/backrest 1.14.1) + restic 0.19.1, three instances (iris, metis, handtrixxx.com), all pairing into the iris dashboard.
  • Destination: Deco router SMB share //192.168.68.1/G (NTFS, 12.7 TiB, auth off) mounted SMB1 only: mount -t cifs //192.168.68.1/G /mnt/deco -o vers=1.0,guest. SMB2/3 negotiation fails (error 22) - it is the router, not the client (a Samba container mounted fine on the same kernel). cifs-utils installed on iris and metis; mounts live but not yet in fstab.
  • Repos: /mnt/deco/restic/{metis,iris,handtrixxx} - one restic repo per host on one spinning HDD (the noise in the room is any of them writing).
  • Transport: metis and iris mount the Deco directly on the LAN; handtrixxx.com has no LAN route and goes through a rest-server container on iris -> rest:http://100.64.0.3:8000/handtrixxx/ (DISABLE_AUTHENTICATION=1). SFTP is dead: restic’s SFTP backend chmods lock/temp files and NTFS-on-SMB1 rejects chmod. Subnet sharing was rejected (extra NAT/MTU risk, Deco exposed to the datacenter).
  • Compose projects /data/docker/backrest/compose.yaml on each host, UI on :9898, auth disabled, LAN/tailnet only. Dashboard hub: hestia http://192.168.68.54:9898 (LAN) / http://100.64.0.4:9898 (tailnet), container backrest-hub. Each of metis, iris, handtrixxx.com runs its own Backrest client (container backrest-backrest-1, UI on its own :9898) that owns that host’s repo and plans.
  • Dashboard hub moved from iris to hestia (2026-10-23, Niklas: option A). Reason: iris becomes the kiosk and is busy; hestia holds the repos anyway. The hub owns no repo or plan, it only collects the operation history of the clients (PERMISSION_READ_OPERATIONS, scope *); every host keeps scheduling its own backup, forget, prune and check (so nothing about the repos or the schedules changed, and the backup data was never touched). Layout: /data/docker/backrest/{compose.yaml,config,data} on hestia, instance id hestia, config seeded with only instance and auth.disabled, data on the SD card (small, about 1.4 MB). Migration steps, for a rebuild: (1) bring up the hub container with the minimal config; (2) per client: stop its Backrest, add the hub to sync.knownHosts with instanceId, keyId, instanceUrl and initialPairingSecret from a fresh token (POST /v1.Backrest/GeneratePairingToken on the hub, body {"ttlSeconds":1800,"maxUses":1,"permissions":[{"type":"PERMISSION_READ_OPERATIONS","scopes":["*"]}]}; the token looks like <keyid>:<secret>#<instanceid>), start it again; (3) the first connect of each client logs unknown peer and no pairing secret once and succeeds on the retry a minute later; (4) remove the old hub from knownHosts and clear authorizedClients on the client that used to be the hub. Instance URLs: metis and iris use the LAN http://192.168.68.54:9898, handtrixxx.com the tailnet http://100.64.0.4:9898. iris is a plain client now and still backs up its own /data to rest:http://192.168.68.54:8000/iris/ (plan iris-data, 03:00). The old iris hub identity was not copied (every host keeps its own key). Configs were backed up on each host as config.json.bak-pre-hestia-hub-* and config.json.bak-pre-drop-iris-hub-*. Verified 2026-10-23 13:33 UTC: all three hosts connected, 68 operations on the hub, backup-farm-status lists all plans ok (new default URL http://100.64.0.4:9898), a manual iris-data run reported through the new hub, hub RSS about 15 MiB.
  • Plans: iris iris-data 03:00 (keep daily 30, prune monthly <=8% unused, weekly structure-only check); metis metis-data 04:00 (+ forget + index); handtrixxx handtrixxx-data 23:00, handtrixxx-sources 23:20, handtrixxx-apps 23:40, handtrixxx-media 00:00 (was 20:00, prime time; all moved 2026-10-23 so the small plans finish before media), check every 14 days; hestia hestia-data 05:00 (own repo hestia, run by the hub on hestia, /data mounted read-only at /backup/data, keep daily 30). All --pack-size 32; ZFS excludes **/.zfs/**.
  • ZFS snapshots still taken by sanoid on metis rpool/data (/etc/sanoid/sanoid.conf: daily=3, timer every 15 min) - snapshots are the source, restic the off-box copy. Inside the container there is no zfs binary, so snapshot hooks are NOT implemented; live reads are the consistency model.
  • Source sizes (measured 2026-10-19): metis /data 254 GB (docker 196, sources 59); iris /data 4.3 GB; handtrixxx.com /data 51 GB + /zfsdata/apps 502 GB + /zfsdata/media 3.1 TB + /zfsdata/sources 3 GB (excluded: zfsdata/backup, downloads, temp). Total ~3.9 TB.
  • Seed progress: metis first run 2026-10-19 20:13 UTC 267.3 GiB (snapshot 07066092), second 2026-10-20 04:00 UTC 267.3 GiB (snapshot 65242153). handtrixxx media handtrixxx-media (~3.1 TiB / 45,657 files of game ROMs/ISOs under /backup/zfsdata/media) started 2026-10-20 05:24 UTC, ~22 MB/s. Snapshot as of 2026-10-20 16:09 UTC (from the ThinkPad): media 22.2% (698.5 GiB), apps ok (464.4 GiB stored, 261,988 files), data/sources ok, metis-data ok, iris-data plan FAILED at 2026-10-20 03:00 UTC - cause not yet investigated. Operation status 3 in data/oplog.sqlite = completed successfully.

Checking backup health

Helpers in kb bin/ (after mem sync every host has them; ThinkPad path is /home/handtrixxx/code/ai-persistence/bin/...):

  • **bin/backup-status** - per-host view: container state, last scheduled/run/failed tasks, repo snapshots, live progress if a backup is running on that host.
  • **bin/backup-overview** - global from any server: reads the hestia hub’s process log over ssh handtrixxx@192.168.68.54 (every peer streams progress there) and prints every RUNNING backup across metis/iris/handtrixxx with percent, MB/s and ETA (it ignores log lines that carry an operations count instead of a list, fixed 2026-10-23).
  • **bin/backup-farm-status** - global via the iris HTTP API, no docker/sudo needed (works from the ThinkPad): per-plan running progress or last result; --plan handtrixxx-media, --json, --url. Raw commands:
docker compose -f /data/docker/backrest/compose.yaml ps            # container Up
docker logs backrest-backrest-1 --since 2026-10-19T00:00:00        # schedule + runs
sudo tail -f /data/docker/backrest/data/processlogs/backrest.log   # per-operation log (root-owned)
sudo sh -c 'P=$(python3 -c "import json;print(json.load(open(\"/data/docker/backrest/config/config.json\"))[\"repos\"][0][\"password\"])"); docker exec -e RESTIC_PASSWORD="$P" backrest-backrest-1 restic -r /restic/metis snapshots'

The Backrest UI does not expose a live run log; the process log is the source. Raw API: POST /v1.Backrest/GetOperations {"selector":{}} - running ops have operationBackup.lastStatus.status.percentDone.

Gotchas (verified)

  • Host crash kills a running backup (2026-10-21): handtrixxx.com hard-reset at 03:23 CEST (no shutdown messages in the previous boot’s journal, cause unknown); the 20 h handtrixxx-media run (39%, 1.32 of 3.38 TB) ended as STATUS_ERROR "Operation was incomplete when orchestrator was restarted". restic writes a snapshot only at the end, so no media snapshot existed; the startup prune/check also failed with “repo already locked” (cleared on its own, restic list locks empty). Recovery = just re-run the plan; restic skips already-uploaded data. Manual start without the UI: ssh handtrixxx.com 'curl -s -m 20 -X POST -H "Content-Type: application/json" -d "{\"value\":\"handtrixxx-media\"}" http://127.0.0.1:9898/v1.Backrest/Backup' (curl times out/returns HTTP 000 but the run keeps going; follow with bin/backup-farm-status --plan handtrixxx-media). Restarted 2026-10-21 06:08 UTC. One repo = one job at a time, so the 02:00 handtrixxx plans queue behind a running media backup.
  • CRITICAL compose gotcha: the backed-up path must be mounted INTO the container or restic silently backs up the container’s own /data (this actually happened on the first iris run). Convention: - /data:/backup/data (read-write so restores land on the host) and - /zfsdata:/backup/zfsdata; plan paths use /backup/.... The container runs as root, so root-owned docker volume files are readable - the “restic needs root” problem solved by Docker.
  • Stale restic locks: iris forget failed 2026-10-19 22:13 UTC with “repository is already locked by PID …” - the locking process died and left orphan lock files. Fixed 2026-10-20 08:24 CEST with restic unlock inside the backrest container (iris: -r /restic/iris, 2 locks removed; handtrixxx: -r rest:http://100.64.0.3:8000/handtrixxx/, 1 lock removed). unlock only drops stale locks; a running backup’s lock is renewed and untouched. Repo password lives in each host’s /data/docker/backrest/config/config.json - do not print. Recurred 2026-10-23 (all four handtrixxx plans showed FAILED in bin/backup-farm-status): the backups themselves had run and written snapshots (05:39-05:43 UTC, after the catch-up of missed runs), only the follow-up forget task failed with unable to create lock in backend ... locked by PID 143 on <container id> ... created 2026-10-21 13:49 - an orphan lock from the 2026-10-21 reboot/shutdown of the servers (the backrest container had been killed mid-run; the lock came from the old container PID, same host id). Fixed 13:11 UTC: restic -r rest:http://100.64.0.4:8000/handtrixxx/ unlock inside backrest-backrest-1 (1 lock removed). Diagnose with the container log (docker logs backrest-backrest-1 | grep 'task failed') and restic list locks. After the unlock all four plans were re-run by hand at 13:15 UTC (incremental, 2-60 s each) and backup-farm-status showed all ok; metis and iris repos had no stale locks and no failed tasks. Trigger a plan manually (on the host that runs it, the UI port is published): curl -X POST -H 'Content-Type: application/json' -d '{"value":"handtrixxx-sources"}' http://127.0.0.1:9898/v1.Backrest/Backup (returns {} at once, the run is async; watch docker logs backrest-backrest-1). After any host shutdown or crash, check locks on all three repos (restic -r <repo> list locks in the backrest container) and run the plans once. Note the container is named backrest-backrest-1 on handtrixxx.com, not backrest.
  • Multihost pairing: POST /v1.Backrest/GeneratePairingToken (permissions as objects: {"type":"PERMISSION_READ_OPERATIONS","scopes":["*"]}); clients put the secret in sync.knownHosts[].initialPairingSecret plus server keyId/instanceUrl in config.json; iris then lists metis+handtrixxx under sync.authorizedClients. “pairing failed: unknown peer and no pairing secret provided” (2026-10-19) means pairing is incomplete - harmless for local backups, worth re-pairing.
  • API casing: file download = POST /v1.Backrest/GetDownloadURL {"opId":N,"filePath":"/backup/..."} (camelCase) -> signed ./download/<jwt> URL, byte-identical diff verified. Restore = POST /v1.Backrest/Restore {"planId":...,"repoId":...,"snapshotId":...,"path":...,"restorePath":...} (planId required); restore target is a path inside the container, so restoring to /backup/... writes to the host. restic ls -l shows uid/gid/mode/symlinks intact.
  • restic stores Unix permissions in its index, immune to NTFS/CIFS permission mangling. Encryption off for now (decided 2026-10-19, Niklas); restic supports it if the share ever gets auth.
  • prune is IO heavy: monthly, 5-10% unused threshold; RESTIC_PACK_SIZE=32 (default 16 MiB) for a TiB-scale repo on a spinning HDD. Must be set for every command that writes to the repo.
  • Peer sync retries (the hub is hestia now) / no dedicated watcher script on metis: monitoring is the commands above plus the Backrest UI. /tmp/restic.bz2 (10 MB, 2026-10-19) was the restic binary download from setup.

Measured throughput (2026-10-19, raw ssh -o Compression=no dd pipe)

  • handtrixxx.com -> iris (tailnet): 20.6 MB/s single stream; 4 parallel streams = 23.6 MB/s aggregate -> parallelism buys nothing, the WAN path is the ceiling (~160-190 Mbit/s). restic’s default sftp.connections=5 is pointless here and costs iris CPU.
  • handtrixxx.com -> metis (tailnet): 20.0 MB/s -> iris’s 2 cores are not the bottleneck.
  • LAN host -> Deco: ~30-32 MB/s write / 32 MB/s read (sha256 verified after remount).
  • Seed estimate: handtrixxx ~3.7 TB at 20 MB/s = ~50 h (restic resumes interrupted runs from its partial index); metis 254 GB over LAN ~2.5 h; iris minutes.

Alternatives considered (rejected)

ZFS send streams (fastest but un-browsable - kills single-file restore), borg (runner-up), plain rsync (loses permissions/retention). Backrest covers the UI criteria: snapshot browser + single-file download, restore-to-path, cron/interval scheduling, time-bucketed retention, scheduled check/prune, operation tree with logs, hooks (Discord/Slack/Gotify/Healthchecks/command - no SMTP), multihost sync. Do not write backup logic in Rust (see parasanoid).

Source: Wiki.js