niklas‑stephan.de

Homelab Server Inventory

Last updated 05/11/2026 · migrated-from-outline

.

The homelab has 3 servers (handtrixxx.com, metis, iris) plus the backup Pi hestia (a 4th host, storage-only; joined the SSH mesh 2026-10-27), plus Niklas’s ThinkPad dev laptop (a full SSH-mesh member since 2026-10-11) and multiple clients. Each server hosts the /data workspace, they are connected via Tailscale for particular services. All run Linux and focus on providing dockerized services only (the laptop is a dev client, not a service host).

Access matrix and sync (verified 2026-10-14, merged from the former host-topology)

  • Tailscale/headscale addresses are stable: metis 100.64.0.1, iris 100.64.0.3, hestia 100.64.0.4; LAN 192.168.68.x (hestia 192.168.68.54). All hosts: user handtrixxx, key-based ssh.
  • handtrixxx.com’s sshd listens on port 2222 (metis and iris have a Host handtrixxx.com stanza in ~/.ssh/config).
  • From handtrixxx.com: ssh metis, ssh iris. From metis: ssh iris, ssh handtrixxx.com, ssh hestia. From iris: ssh metis, ssh handtrixxx.com, ssh hestia. hestia joined the SSH mesh 2026-10-27: keys of handtrixxx.com, metis, iris and the ThinkPad were added to its authorized_keys (Niklas’s Mac key niklas.stephan@DE08-M0040 was already there), and a new keypair (comment handtrixxx@hestia) was generated on hestia and authorized on all four other hosts. ssh hestia aliases: LAN 192.168.68.54 on metis/iris/ThinkPad, tailnet 100.64.0.4 on handtrixxx.com. Bidirectional key auth verified from every host. hestia still has password login enabled deliberately (Niklas’s choice). ThinkPad’s inbound lockdown (/etc/ssh/sshd_config.d/60-homelab-only.conf) was extended with handtrixxx@100.64.0.4 handtrixxx@192.168.68.54 so hestia can SSH into it.
  • Every host has /data/AGENTS.md (generated) and the memory repo /data/ai-persistence, synced via HTTPS to Gitea (git.handtrixxx.com, container githandtrixxxcom-server-1, SSH disabled). metis and iris have no GitHub credentials.
  • handtrixxx.com additionally runs Outline (read-only archive now), Traefik, Matomo etc. under /data/docker.

Server “Metis”

  • Location: homelab local LAN
  • Hardware: Ubuntu 26 x86_64, 32-core AMD (Ryzen AI MAX+ 395), 64GB RAM unifiied, GPU (Radeon 8060S), 647GB ZFS root. Most powerful box; runs local LLM inference.
  • Network: 192.168.68.50 (metis.local)
  • SSH: port 22, user handtrixxx (uid 1001)
  • Role: headless docker host - the desktop environment (GNOME/Xfce/LXQt/X, kiosk autostart) was removed 2026-10-20, boot target is multi-user.target; details in metis-headless-server. Home Assistant moved from here to iris on 2026-10-21 (the stopped container is kept as rollback); the Home Assistant touchscreen kiosk runs on iris. metis keeps the LLM server and Wolf (headless game streaming, brought back up 2026-10-25 per Niklas). The voice stack (whisper/piper/openwakeword) moved to iris on 2026-10-23.
  • Main docker projects (verified 2026-10-26, five dirs in /data/docker, all deployed): llama (llama-flashnext, :8000), tailscale (tailscale-metis), backrest (backup client, UI :9898), wolf (wolf-wolf-1, Moonlight :47999), arcane-agent (:3553, Arcane remote agent added 2026-10-26). Details in docker-projects-on-metis.
    • Urbackup client: removed 2026-10-19; HomeAssistant MOVED to iris 2026-10-21 (dir deleted 2026-10-23); local.ai is an old hostname for the llama server, not a project name

Server “Iris”

  • Location: homelab local LAN
  • Hardware: Ubuntu 26 x86_64, 2-core Intel (Pentium G4400), 16GB RAM, iGPU, 436GB ZFS root (the 14TB backup disk was moved to the Deco router’s USB port on 2026-10-19 and on 2026-10-21 to the Raspberry Pi hestia, see pi-backup-server). Low-power box; avoid heavy workloads.
  • Network: 192.168.68.63 (iris.local)
  • SSH: port 22, user handtrixxx (uid 1000)
  • Main docker projects:
    • HomeAssistant — home automation (moved here from metis 2026-10-21; ConBee II Zigbee stick plugged in; touchscreen kiosk runs against it)
    • MusicAssistant — music streaming
    • Adguard — DNS server and filter for the homelab
    • Frigate — camera recordings
    • MQTT broker — for Frigate → HomeAssistant
    • voice — whisper/piper/openwakeword (Wyoming), moved from metis 2026-10-23, ports bound to 127.0.0.1 only (HA is on the same box)
    • backrest — Backrest client for the iris restic repo (UI :9898)
    • Tailscale client — a container (tailscale/tailscale, host network), not a host-installed tailscaled; also serves the SOCKS5 proxy :1055 for metube egress and runs 4 socat forwarders to handtrixxx.com services
    • Metube — video downloader, host-networked, egresses via the tailscale exit node (moved here from handtrixxx.com 2026-10-04)
    • Urbackup server and client: RETIRED 2026-10-19, containers/images/project dir deleted 2026-10-21 (see backup-scenario-restic) Full table with ports: memory docker-projects-on-iris

Server “Hestia” (Raspberry Pi 5, backup server, set up 2026-10-21)

  • Hardware: Raspberry Pi 5, 8 GB RAM, 128 GB SanDisk A2 microSD, Ubuntu Server 26.04.1 LTS (aarch64). Next to the Deco router, wired. The 14 TB WD Elements (WD140EDGZ, ext4 at /srv/backup) is attached to its USB 3 port.
  • Network: LAN 192.168.68.54 (DHCP reservation set in the Deco app by Niklas), tailnet 100.64.0.4. SSH port 22, user handtrixxx, full mesh keys installed (2026-10-27), sudo NOPASSWD. Reachable from metis, iris and handtrixxx.com.
  • Software: Docker 29.9.2 + Compose v5.6.1 from Docker’s apt repo (docker.sources, not Ubuntu’s docker.io), /data/docker/ layout like the other servers, tailscale host-network container /data/docker/tailscale/ (headscale user homelab-tunnel, key TS_AUTHKEY_HESTIA in its .env). journald capped at 100 MB.
  • Role: restic rest-server (container restic-server, port 8000, auth off, repos in /srv/backup/restic) for all backups; plan and status in memory pi-backup-server. Compose projects verified 2026-10-25: restic-server (:8000), backrest (Backrest hub UI, backrest-hub, :9898), tailscale (tailscale-hestia), arcane-agent (added 2026-10-26, :3553, managed from docker.handtrixxx.com - see arcane-remote-environments) - nothing else runs on the Pi.

Server “handtrixxx.com”

  • Location: remote datacenter
  • Hardware: Ubuntu 24 x86_64, 12-core Intel (i7-8700), 64GB RAM, iGPU, 436GB ZFS root + 12TB ZFS data disk (/zfsdata).
  • Network: 148.251.15.253 (handtrixxx.com)
  • SSH: port 2222, user handtrixxx (uid 1000)
  • Main docker projects:
    • Headscale server (headscale.handtrixxx.com) + host-level Tailscale client (/data/docker/tailscale/, node handtrixxx = 100.64.0.2, added 2026-10-04; the per-project sidecars ai/homelab.vpn/urbackup-client were retired in the same consolidation)
    • Traefik — reverse proxy
    • Stalwart — email server
    • Other services

Dev laptop “handtrixxx-ThinkPad” (full SSH-mesh member since 2026-10-11)

Niklas’s daily driver: chats with Hermes via the desktop app and is the main dev machine. Onboarded 2026-10-11 as a full member of the homelab stack — not just a client.

  • Location: homelab local LAN (also reachable everywhere over the tailnet)
  • Hardware: Lenovo ThinkPad L13 2-in-1 Gen 5 (convertible), Intel Core Ultra 5 135U (14 threads), 32GB RAM, 837GB NVMe (/, ~533GB free)
  • OS: Ubuntu 26.04.1 LTS (kernel 7.0.0-1005-gcp), user handtrixxx (uid 1000)
  • Network: LAN 192.168.68.66/22 + tailnet 100.64.0.8 (node handtrixxx-thinkpad, host-level tailscaled, user homelab-tunnel)
  • SSH: port 22, user handtrixxx. openssh-server installed + ssh service enabled 2026-10-11 (it was not installed before — sshd/ssh unit didn’t exist, so the package is openssh-server, not sshd). ~/.ssh/authorized_keys holds the pubkeys of handtrixxx.com, metis, iris and (since 2026-10-27) hestia, so all of them can key-auth in (verified 2026-10-11 / 2026-10-27).
  • Inbound lockdown (2026-10-11): /etc/ssh/sshd_config.d/60-homelab-only.conf sets PasswordAuthentication no + AllowUsers handtrixxx@<ip> for the homelab hosts’ source IPs (extended 2026-10-27 with hestia 100.64.0.4/192.168.68.54; this lockdown does not exist on hestia itself, which keeps password login on purpose) (tailnet 100.64.0.1/.2/.3/.4 + LAN 192.168.68.50/.63/.54). Only those may SSH in, key auth only, as handtrixxx — all other sources/users/passwords are denied. Verified via sshd -T (effective config) + key auth from all 3 servers.
  • Outbound SSH: already had ~/.ssh/config aliases for metis (192.168.68.50), iris (192.168.68.63), and handtrixxx.com (port 2222) — laptop → all three servers worked from the start.
  • Role: dev client — no hosted docker services, no /data workspace, no AGENTS.md. Verified 2026-10-25: no /data/docker here; the only containers are a buildx_buildkit_xarch0 cross-arch builder (up) and an exited Alexandria debug container alx-dbg2. Treated as a peer in the SSH mesh (servers ↔ laptop) and a tailnet node, not a service host.
  • Note: full passwordless sudo for handtrixxx since 2026-10-11 (/etc/sudoers.d/handtrixxx-nopasswd, same convention as the servers; verified over SSH). Not part of the /data AGENTS.md sync set (no /data workspace).

SSH mesh (3 servers + ThinkPad laptop + hestia since 2026-10-27, key auth)

Each server has a ~/.ssh/config and its own id_ed25519 key (comment = handtrixxx@<host>). All three pubkeys are authorized in all three boxes’ authorized_keys, so any server can key-auth to any other where the network allows (see table below).

Config pattern per host (Metis example; Iris is identical but lists metis + handtrixxx.com; handtrixxx.com uses IPs for iris/metis because it can’t resolve .local):

Host iris iris.local
    HostName iris.local        # on handtrixxx.com use the tailnet IP 100.64.0.3 (no LAN route)
    User handtrixxx
    Port 22
    IdentityFile ~/.ssh/id_ed25519
    IdentitiesOnly no

Host handtrixxx.com
    HostName handtrixxx.com
    User handtrixxx
    Port 2222                  # NOT 22
    IdentityFile ~/.ssh/id_ed25519

Verified connectivity (2026-10-11, ThinkPad added)

From \ To iris metis handtrixxx.com thinkpad
metis ✅ key auth — ✅ key auth ✅ key auth (tailnet)
iris — ✅ key auth ✅ key auth ✅ key auth (tailnet)
handtrixxx.com ✅ via tailnet ✅ via tailnet — ✅ key auth (tailnet)
thinkpad ✅ (LAN) ✅ (LAN) ✅ (tailnet) —
  • ThinkPad 100.64.0.8 reached from the servers over the tailnet (all three verified key auth 2026-10-11); the laptop reaches metis/iris over the LAN and handtrixxx.com over the tailnet (its ~/.ssh/config aliases work — verified 2026-10-11). The laptop’s authorized_keys holds the three servers’ pubkeys (handtrixxx@handtrixxx.com, handtrixxx@metis, handtrixxx@iris).
  • handtrixxx.com → iris/metis over the LAN (192.168.68.x) times out at the network layer — the datacenter has no route to the homelab LAN. Since 2026-10-04 handtrixxx.com runs a host-level tailscaled (sidecars retired), so SSH over the tailnet works. The ~/.ssh/config aliases on handtrixxx.com were repointed (2026-10-04) to the tailnet IPs — Host iris ... 100.64.0.3 / HostName 100.64.0.3 and Host metis ... 100.64.0.1 / HostName 100.64.0.1 — so plain ssh iris / ssh metis now work (verified). Backup of the pre-change config: ~/.ssh/config.bak-tsaliases.
  • Tailnet IPs (since 2026-10-04): metis 100.64.0.1, handtrixxx 100.64.0.2, iris 100.64.0.3 — all three servers are host-level clients; thinkpad 100.64.0.8. Full map in the Headscale / Tailscale Tailnet Reference doc.

/data workspace & AGENTS.md sync (metis + iris)

  • All three boxes (metis, iris, handtrixxx.com) have /data/AGENTS.md — identical except line 3, which records the local hostname (This server is metis. / ...iris. / ...handtrixxx.com.). Keep them in sync so it doesn’t matter which box you connect to. (handtrixxx.com already had one; only normalized its missing trailing period on 2026-10-04.)
  • /data must be owned by handtrixxx:handtrixxx for the agent (and user) to write the workspace non-interactively. Gotcha: on iris /data was originally root:root, and the handtrixxx user there has no passwordless sudo — so the agent hit Permission denied writing /data/AGENTS.md. Fixed 2026-10-04 with a one-time (user-run) sudo chown handtrixxx:handtrixxx /data. Verify any new box with: ssh <host> "ls -ld /data; touch /data/.t && echo OK && rm /data/.t".
  • To sync AGENTS.md between boxes without shell-quoting pain (the content has backticks that break heredocs-over-ssh): scp /data/AGENTS.md iris:~/AGENTS.md then sed -i 's/^This server is metis\.$/This server is iris./' ~/AGENTS.md, then move into /data.

Passwordless sudo (all 4 hosts)

  • All four boxes (metis, iris, handtrixxx.com, and the ThinkPad since 2026-10-11) have full passwordless sudo for handtrixxx via /etc/sudoers.d/handtrixxx-nopasswd → handtrixxx ALL=(ALL) NOPASSWD: ALL (mode 440, root:root). Verified 2026-10-04: sudo -n true passes on metis, iris, and handtrixxx.com; sudo visudo -c parses OK everywhere.
  • So the agent can run privileged ops non-interactively on any box (e.g. ssh iris "sudo zfs list", install packages, chown, etc.) — no password prompts.
  • Note: handtrixxx.com also has a pre-existing scoped rule /etc/sudoers.d/syncoid (NOPASSWD: /sbin/zfs) for the syncoid replication job; leave it alone.

Gotchas

  • The Zed agent shell has no TTY and no forwarded SSH agent, so it can only use keys stored on the local box — not a laptop key forwarded into an interactive login. If ssh <host> works in your terminal but fails for the agent, the cause is almost always “this box’s own key isn’t in that host’s authorized_keys”. Fix by appending this box’s pubkey there.
  • To authorize one box’s key on another from a box you can already reach: ipub=$(ssh iris 'cat ~/.ssh/id_ed25519.pub'); ssh handtrixxx.com "grep -qF \"\$ipub\" ~/.ssh/authorized_keys || echo \"\$ipub\" >> ~/.ssh/authorized_keys".
  • (Obsolete since 2026-10-19: the usbbackup pool is gone, so Iris df -h is no longer flooded by Urbackup snapshot datasets.)
  • OpenSSH **Match** has NO **!**-negation (learned the hard way 2026-10-11): Match Address !1.2.3.4 is rejected (Unsupported Match attribute ... / Bad Match condition), and because sshd aborts on an invalid config, systemctl restart with such a drop-in kills the running daemon and leaves the box SSH-unreachable (fix only from the local console). To restrict inbound SSH by source, use AllowUsers user@ip lists instead (always sudo sshd -t before restarting). Applied on the ThinkPad as /etc/ssh/sshd_config.d/60-homelab-only.conf.

OpenClaw gateway -> homelab SSH (REMOVED 2026-10-23)

OpenClaw was shut down and deleted on handtrixxx.com on 2026-10-23; its SSH key openclaw-gateway@handtrixxx.com was removed from authorized_keys on metis, iris and handtrixxx.com. HASS_TOKEN stays in /data/ai-persistence/.env on each server.

Source: Wiki.js