Headscale / Tailscale Tailnet Reference

Last updated 05/11/2026 · migrated-from-outline, docker, tailscale, headscale

.

The homelab runs its own Tailscale control server (Headscale) on handtrixxx.com at https://headscale.handtrixxx.com. All servers and service sidecars join this tailnet instead of using tailscale.net. Project: /data/docker/headscale.handtrixxx.com/ (README there has the full setup template).

Current deployment (handtrixxx.com)

• Container headscale, image headscale/headscale:latest, command serve
• Web/API on 8080 behind Traefik (headscale.handtrixxx.com, websecure entrypoint, existing cert); DERP/STUN on UDP 3478 (published directly)
• State in ./config (config.yaml, acl.hcl) and ./data (sqlite db, keys)
• Tailnet prefix: v4 100.64.0.0/10 + v6 fd7a:115c:a1e0::/48; own DERP region 999 “Hetzner Self-Hosted” (no upstream Tailscale DERP used, urls: [], auto_update_enabled: true); magic DNS off (base domain ts.handtrixxx.com)
• ACL policy mode: file (/etc/headscale/acl.hcl)

Tailnet node map (verified 2026-10-04)

All nodes belong to the single user **homelab-tunnel@** — there is no per-user split. metis, iris, and handtrixxx.com are full host-level clients (network_mode: host, each advertises an exit node, custom port 41641). The per-project sidecars on handtrixxx.com (ai, hetzner-homelab, hetzner) were retired and deleted on 2026-10-04 — see the consolidation report.

Iris forwarders: iris’s /data/docker/tailscale/ also runs 4 socat containers (forward-calibre/jellyfin/sabnzbd/nzbhydra, host networking) that bridge LAN ports → 100.64.0.2 (the homelab.vpn apps on handtrixxx.com; targets updated 2026-10-04 when the sidecar IP 100.64.0.5 was retired).

metube moved to iris (2026-10-04): was the metube.handtrixxx.com sidecar (metube-egress, 100.64.0.2, now deleted from headscale). Now a plain host-networked container in /data/docker/metube/ on iris; yt-dlp egresses through iris’s tailscale SOCKS5 proxy (socks5h://localhost:1055) → metis exit node (residential IP). The public metube.handtrixxx.com Traefik URL was dropped — reach it at http://iris:8081 (LAN) or its tailnet IP.

Access a service example: curl -f http://100.64.0.2:8083/login (Calibre-Web on handtrixxx.com, over the tailnet).

Setting up Headscale on a fresh server (from README)

Compose: image headscale/headscale:latest, command: serve, volumes ./config:/etc/headscale + ./data:/var/lib/headscale, port 3478:3478/udp, on the caddy network with Traefik labels (router → service port 8080, websecure entrypoint).

config/config.yaml essentials:

server_url: https://headscale.handtrixxx.com
listen_addr: 0.0.0.0:8080
prefixes: { v4: 100.64.0.0/10 }
derp:
  server: { enabled: true, region_id: 999, stun_listen_addr: "0.0.0.0:3478" }
  urls: []                    # no Tailscale DERP — only your own
  auto_update_enabled: true
dns: { magic_dns: false, base_domain: ts.handtrixxx.com }
database: { type: sqlite, sqlite: { path: /var/lib/headscale/db.sqlite } }
policy: { mode: file, path: /etc/headscale/acl.hcl }

config/acl.hcl: in practice a single user (homelab-tunnel@) holds all nodes. The live policy is two accept rules for that user (to all its own nodes USER@:*, and to autogroup:internet:* for egress) plus autoApprovers.exitNode: ["homelab-tunnel@"]. When you do add a second user, mirror the same pattern per user:

{
  "acls": [
    { "action": "accept", "src": ["USER@"], "dst": ["USER@:*"] },
    { "action": "accept", "src": ["USER@"], "dst": ["autogroup:internet:*"] }
  ],
  "autoApprovers": { "exitNode": ["USER@"] }
}

Client (sidecar) pattern

Legacy pattern (superseded 2026-10-04 — all servers now run host-level clients): each project that needs the tailnet runs a tailscale/tailscale sidecar:

environment:
  - TS_AUTHKEY=${TS_AUTHKEY_...}          # preauthkey, created with expiration
  - TS_STATE_DIR=/var/lib/tailscale
  - TS_HOSTNAME=<node-name>
  - TS_EXTRA_ARGS=--login-server=https://headscale.handtrixxx.com
  - TS_USERSPACE=false
  - TS_ACCEPT_DNS=false
volumes: [ ./tailscale-state:/var/lib/tailscale, /dev/net/tun:/dev/net/tun ]
cap_add: [NET_ADMIN, NET_RAW]

Apps then use network_mode: "service:<sidecar>". For app-level egress only (no kernel access), skip the caps and use TS_SOCKS5_SERVER=:1055 + TS_AUTH_ONCE=true. (metube no longer uses a sidecar — it’s host-networked on iris and rides iris’s own client SOCKS5 proxy; see “metube moved to iris” above.)

Headscale CLI cheat sheet (run on handtrixxx.com)

docker exec headscale headscale users list                          # current users
docker exec headscale headscale nodes list                          # current nodes
docker exec headscale headscale users create USERNAME               # new user
docker exec headscale headscale preauthkeys create --user 2 --expiration 4h   # client key (short-lived! CLI default is 1h)
docker exec headscale headscale preauthkeys create --user 2 --reusable --expiration 876000h  # long-lived host-client key (no "never" mode exists)
docker exec headscale headscale auth register --auth-id xxx --user homelab-tunnel  # finish web-UI registration
docker exec headscale headscale nodes delete --identifier 1 --force # remove a node

Gotchas

• Preauthkey expiry bites repeatedly: an expired key makes tailscale up fail with invalid pre auth key. Recreating an existing client re-runs tailscale up, so a stored-but-since-expired key crash-loops a previously healthy container. preauthkeys create has no unlimited mode: CLI default --expiration 1h, --expiration 0s = expired at creation. Long-lived host clients: --reusable --expiration 876000h. Live keys since the 2026-10-04 cleanup: IDs 16/17/18 (expire 2126) in each host’s /data/docker/tailscale/.env (TS_AUTHKEY_METIS / TS_AUTHKEY_IRIS / TS_AUTHKEY_HANDTRIXXX).
• Env is baked at container creation — editing **.env** does nothing until a recreate. 2026-10-08: handtrixxx’s tailscaled was restarted with an old deleted key and crash-looped (post-mortem in headscale-tailscale-reference). Rule: compare docker inspect <ctr> --format '{{range .Config.Env}}{{println .}}{{end}}' | grep TS_AUTHKEY | cut -c1-30 with .env; finish every key rotation with docker compose up -d --force-recreate and then restart once to prove it survives. The ./state dir preserves node identity (same IP).
• Debugging auth failures: headscale logs at warn, so rejected registrations are silent; the reason appears in the client container log. Key format hskey-auth-<12 chars>-<64 chars> (bcrypt-hashed; dashes inside prefix/secret are legal, parsed by fixed length).
• Traefik v3.7 + host-net containers: the docker provider resolves host-net containers to 127.0.0.1 (Traefik’s own netns) -> 502, masked by the catch-all 301. loadbalancer.server=<url> in a docker label is invalid (struct node -> invalid slice: node server, the container’s whole config is silently dropped); service=x@file in a docker label is not honored. Working pattern: router + service in the file provider (/data/docker/traefik.handtrixxx.com/config/dynamic/host-net.yml, watch: true) with url: http://172.19.0.1:<port> (caddy gateway), priority 20; host-net containers carry zero traefik labels (exposedByDefault: false). Never put a YAML comment inside a quoted label value.
• Host-net containers have no docker DNS — sibling container names (e.g. gluetun) don’t resolve from host net. gluetun’s IP on homelab_tunnel is pinned to 172.22.0.2 since 2026-10-09 (external network 172.22.0.0/16 accepts static IPs); after any gluetun/nzb.vpn recreate verify the pin and the proxy-sabnzbd/proxy-nzbhydra2 socat targets.
• Hetzner Robot firewall is stateless (max 10 rules/direction, unmatched packets dropped). Inbound UDP replies (DNS/53, NTP/123, STUN) were silently blackholed until rules #9 (udp src 53 -> 32768-65535 accept) and #10 (udp src 123 -> 32768-65535 accept) were added 2026-10-09. All 10 inbound slots are now used - freeing one requires a review. Server has no global IPv6. DNS therefore runs over DoT (quad9/cloudflare via systemd-resolved); tailnet UDP to the host works because 3478/41641 are explicitly allowed.
• Forwarder health checks: nc -z only proves socat listens locally (the dead forward-adguard target looked healthy). Use wget -q -T 5 -O /dev/null http://127.0.0.1:<port>/; Jellyfin needs /health (its / 302-redirects to a host BusyBox wget cannot follow). iris forwarders use the wget check; the homelab.vpn forwarders on handtrixxx.com still use nc -z (known weakness, tracked in homelab-backlog). Forwarder targets: adguard -> 100.64.0.3:81 (fixed 2026-10-09; 100.64.0.6 never existed).
• DNS on metis: /etc/resolv.conf is a static hand-written file (NM’s resolvconf hook fails), so DHCP/NM DNS changes never apply (open item in the backlog). handtrixxx.com: systemd-resolved with DoT from /etc/systemd/resolved.conf; no MagicDNS (headscale magic_dns: false).
• docker exec needs -i for stdin heredocs; edit Home Assistant / Music Assistant state JSON with the container stopped (it is rewritten on shutdown).
• Open: check iris’s tailscaled for TS_DEBUG_FIREWALL_MODE=nftables + current key. Metis applied 2026-10-09: firewallmode="nft-forced", ts-* chains + masquerade present, exit-node NAT functional; the recreate also removed a stale deleted key that would have crash-looped on the next restart.

Source: Wiki.js