niklas‑stephan.de

Raspberry Pi 5 backup server hestia (live since 2026-10-21)

Last updated 05/11/2026 · backup, restic, raspberry-pi, network

Status 2026-10-21: LIVE (set up the same day)

Replaces the Deco USB share (see backup-scenario-restic, backup-scenario-restic). Niklas has a Raspberry Pi 5; it sits next to the router, wired by cable. The 14 TB WD Elements (own 12 V power adapter, so no Pi power issue) goes on a blue USB 3 port of the Pi.

Progress 2026-10-21

Done: Pi 5 hestia flashed (Ubuntu 26.04.1), on LAN 192.168.68.54, metis + Mac key + NOPASSWD sudo (password SSH login deliberately stays on, Niklas’s choice), Docker CE 29.9.2 + Compose from Docker’s apt repo, /data/docker, tailscale container joined headscale as 100.64.0.4, journald 100 MB cap, unattended-upgrades. Disk: WD140EDGZ attached on USB 3 as /dev/sda, SMART PASSED (0 realloc/pending, 4183 h), old NTFS repos wiped, GPT + ext4 (-L backup -m 0 -T largefile), mounted by UUID at /srv/backup (defaults,noatime,nofail), repos dir /srv/backup/restic (owner uid 1000). rest-server container restic-server in /data/docker/restic-server/compose.yml (image restic/rest-server:latest, user: 1000:1000, port 8000, DISABLE_AUTHENTICATION=1, append-only off). Measured: local disk write 215 MB/s; restic backup through rest-server from metis (LAN) ~90 MB/s, iris (LAN) ~55 MB/s, handtrixxx.com via tailnet (direct path, 42 ms) ~19 MB/s for a 300 MiB test - that is the home upload, not the Pi, so the 3.1 TB media seed will still take roughly 2 days. Test repos deleted.

Backrest on all three hosts repointed to the Pi and first runs triggered (see backup-scenario-restic); iris relay and /mnt/deco removed. Not done: /data/ai-persistence is not cloned on the Pi (no agent workflow there yet). Deco DHCP reservation for 192.168.68.54 was set by Niklas.

Hestia also runs the Backrest dashboard hub (since 2026-10-23)

Container backrest-hub, /data/docker/backrest/ on hestia, UI http://192.168.68.54:9898, auth off, about 15 MiB RAM. It owns no repo; metis, iris and handtrixxx.com pair into it. Full migration notes: backup-scenario-restic. The hub’s state lives on the SD card (small); the backup data on /srv/backup is unaffected.

Target design

  • OS: Ubuntu Server 26.04 LTS 64-bit (decided), flashed with Raspberry Pi Imager (sudo snap install rpi-imager on Ubuntu): preset hostname, user handtrixxx, SSH key. Boot from a good microSD or USB/NVMe; keep journald small or volatile to spare the card.
  • Hostname: not decided (suggestion hestia, matching the Greek names). Give it a DHCP reservation in the Deco app so the LAN IP is stable.
  • Disk: reformat NTFS to ext4 (mkfs.ext4 -L backup -m 0), mount by UUID at /srv/backup with nofail. The old NTFS repos (about 2 TB) are discarded and re-seeded. SMART: smartctl -d sat -a /dev/sdX (printed nothing through the Deco; direct USB on the Pi may work).
  • Service: docker plus one compose project /data/docker/restic-server/ running restic/rest-server with /srv/backup/restic:/data, port 8000. Replaces the iris relay container.
  • Tailscale: install on the Pi and join headscale (--login-server=https://headscale.handtrixxx.com, reusable long-lived preauth key, see headscale-tailscale-reference). handtrixxx.com then writes directly to the Pi’s tailnet IP; the iris restic-server project is removed.
  • Backrest: the three instances keep their plans, only the repo URIs change: metis and iris rest:http://<pi-lan-ip>:8000/<host>/, handtrixxx.com rest:http://<pi-tailnet-ip>:8000/handtrixxx/. The new repos are empty and need restic init (Backrest can init on repo add), then run the plans. UI stays on iris (:9898).
  • Re-seed: estimate 12-15 h wall time at 80-100 MB/s over gigabit, versus 50+ h on the Deco. The 3.1 TB media plan on handtrixxx.com is the long one. Expect uplink to the Pi over the tailnet to be the limit for handtrixxx.com (remote datacenter) - check bandwidth before assuming gigabit.

rest-server facts (official README, checked 2026-10-21)

  • Flags: --append-only, --private-repos, --no-auth, --listen, --path, --max-size, --prometheus. Docker image: data volume is /data (not /tmp/restic), auth is on by default, disabled with env DISABLE_AUTHENTICATION, extra flags via env OPTIONS; users via docker exec -it <name> create_user <user> [pass]; can run non-root with --user and --read-only if /data is writable by that user.
  • Compose sketch: image: restic/rest-server:latest, restart: unless-stopped, ports: ["8000:8000"], volumes: ["/srv/backup/restic:/data"], environment: DISABLE_AUTHENTICATION: "1".
  • --append-only blocks delete and modify, so Backrest **forget**/**prune**/**check --read-data** fail through it. Retention then has to run locally on the Pi, outside the append-only listener.

Decisions 2026-10-21 (Niklas)

  • A: no append-only; Backrest prunes on schedule as before. Auth stays off (DISABLE_AUTHENTICATION=1, LAN and tailnet only).
  • OS: Ubuntu Server 26.04 LTS (same as the other servers; certified on the Pi 5 per ubuntu.com/download/raspberry-pi, checked 2026-10-21). Docker with a /data workspace and /data/docker/<project> layout like the other hosts.
  • Hostname still open (hestia is only a suggestion; nothing depends on it). Boot medium: SanDisk 128 GB A2 V30 microSD (chosen). Flashed from Niklas’s Mac with Imager, password SSH for now; metis’s key still has to be added, then password login switched off.
  • Rejected 2026-10-21: disk back on iris + Pi as HA kiosk (no second network cable near the kiosk screen). The Pi stays the backup server next to the router; the kiosk rebuild stays on iris after the planned position swap.

Open decisions (ask Niklas) - 1 and 2 are settled above, the OS line in 3 is settled

  1. Append-only or retention? Option A (simple, as before): no append-only, Backrest prunes on schedule. Option B (ransomware protection): append-only listener on 8000 for the backups, plus a monthly local restic forget --prune on the Pi (needs repo passwords on the Pi and the Pi’s RAM; a 3 TB repo prune is heavy).
  2. Auth: before, DISABLE_AUTHENTICATION=1 on LAN/tailnet only. Alternative: htpasswd users metis/iris/handtrixxx with --private-repos, the password inside each Backrest repo URI.
  3. Boot medium and hostname (the OS is decided).
  4. After the re-seed: delete the old repos by reformatting; the disk has to be on the Pi first, so there is a gap with no backups until the first plans finish. Metis keeps sanoid ZFS snapshots meanwhile.

Steps (all done 2026-10-21 except the kb clone on the Pi)

  1. Niklas: flash the Pi, connect it, give me SSH access (key for user handtrixxx), attach the disk.
  2. Pi: apt update, unattended-upgrades, docker, tailscale join.
  3. Pi: wipe and format the disk ext4, mount, create /srv/backup/restic.
  4. Pi: start rest-server, verify with restic -r rest:http://<pi>:8000/test init from metis.
  5. Repoint Backrest on iris, metis, handtrixxx.com (edit config.json repos, restart the containers), run the plans, watch with bin/backup-farm-status.
  6. Remove the iris restic-server project and /mnt/deco leftovers; update backup-scenario-restic, homelab-server-inventory, backup-scenario-restic; drop the fstab comment lines.

Gotchas learned during setup (2026-10-21)

  • Raspberry Pi Imager accepts only one SSH public key; add more afterwards (metis’s key was added via ssh with the password).
  • The Pi did not show up on the LAN after the first boot; a reboot made it appear (DHCP lease). Pi 5 MAC prefix 2c:cf:67 helps to spot it in ip neigh.
  • Ubuntu sudo needs the sudoers drop-in /etc/sudoers.d/handtrixxx (NOPASSWD) for agent work; set once by Niklas with ssh -t.
  • restic backup has no --init; run restic init first (Backrest does not auto-init repos that already have a guid). Helper: /data/maintenance/backrest-repoint.py on metis.
  • dd ... iflag=direct can fail on the USB disk; the write test (oflag=direct) gave 215 MB/s.
  • The WD Elements uses the usb-storage driver on the Pi, not UAS; SMART works with smartctl -d sat.
  • Do not write to /srv/backup as non-root: only /srv/backup/restic is owned by handtrixxx (uid 1000 = the restic-server container user).
  • Boot race: container saw an empty **/data** (found 2026-10-21 after the first real reboot). The USB disk mounted ~127 s after boot, but Docker had already started restic-server with the bind mount on the empty directory under /srv/backup, so every repo answered “repository does not exist” (and a backup could have written to the SD card). Symptom to recognise: repos listed on the host (ls /srv/backup/restic) but docker exec restic-server ls /data is empty. Immediate fix docker restart restic-server. Permanent fix (tested with a reboot, mount at 22 s and repos visible): /etc/systemd/system/docker.service.d/wait-backup-mount.conf with [Unit] RequiresMountsFor=/srv/backup, and x-systemd.device-timeout=5min added to the /srv/backup fstab line (backup /etc/fstab.bak-pre-mountwait). Consequence: if the disk is missing, Docker (and tailscale-hestia) will not start; the Pi is then only reachable on the LAN.
  • Backup trigger by hand on a host: curl -m 8 -X POST -H 'Content-Type: application/json' -d '{"value":"<plan>"}' http://127.0.0.1:9898/v1.Backrest/Backup (auth is off; HTTP 000 after 8 s is normal, the job keeps running; 500 means the job failed immediately, read the op message via GetOperations).

Source: Wiki.js