niklas‑stephan.de

weboid — Custom Backup Solution (Architecture)

Last updated 05/11/2026 · migrated-from-outline, weboid

.

Replacement for urbackup (urbackup itself was RETIRED 2026-10-19, see backup-scenario-restic). Repo was at /data/code/weboid on metis; the demo project dir is gone as of 2026-10-19 (verified). weboid remains a candidate for the new backup solution (vs restic/borg/rsync) - the durable design decisions and gotchas below survive for a future rebuild.

Dated session logs from the build (bootstrap UI, SSH discovery, syncoid path, MUI migration) were archived on 2026-10-19; their durable facts are folded into this doc and zfs-syncoid-cli-gotchas.

Why

Unhappy with urbackup; building a centralized backup system from scratch. Requirements (from user):

  • Server + Client Docker images. Client on every backup target; server on the box with backup storage attached.
  • WebUI on the server built with Next.js.
  • Natively support ZFS and btrfs snapshots; rsync (or similar) for the rest.
  • Client container runs privileged to talk to the filesystems.
  • Embed existing tools (sanoid/syncoid, zfstools, btrfs-progs); our code just drives them.

Locked-in decisions (2026-10-04)

  • Language: Go for both binaries (single static binary, easy Docker images, good SSH via golang.org/x/crypto). WebUI stays Next.js/TS.
  • Transport: SSH. Server is always the SSH client; clients open no inbound ports. Per-target keypair; host-key pinning still TODO.
  • State store: file-based JSON under WEBOID_DATA_DIR (one storage box, low write volume). Swap for SQLite/Postgres later without touching callers.

Architecture

Two Go binaries, one repo, two images.

  • weboid-server — on the storage box. Serves Next.js WebUI + JSON API (/api/health|targets|jobs|runs), file-based state, drives clients over SSH. Needs ZFS/btrfs + openssh-server to receive streams.
  • weboid-client — on each target, privileged, no inbound ports. Shells out to the real tools for the local half of a backup.

Transports (one per target)

Kind Local half (client) Server receives via
zfs zfs snapshot → syncoid -s user@server://remote ds@snap syncoid -R
btrfs btrfs subvolume snapshot → btrfs send | ssh … receive btrfs receive
rsync rsync -aHAX --delete path/ user@server:remote/ local dir

Client = privileged side (talks to its own FS); server holds storage + receives. No inbound ports on clients.

Repo layout

weboid/
├── cmd/server/main.go      # API + serves built Next.js
├── cmd/client/main.go      # local daemon, dispatcher for fs runners
├── internal/{types,config,state,transport,fs}/
│   └── fs/                 # Runner interface + zfs/btrfs/rsync impls (stubbed)
├── src/                    # Next.js WebUI (output: "standalone")
├── docker/{server.Dockerfile, client.Dockerfile}
├── compose.yaml            # example server deployment
└── docker/client.compose.yaml  # per-target client (privileged)

Docker notes

  • Gotcha (fixed 2026-10-04): golang:1.23-trixie and node:20-trixie do NOT exist as Docker Hub tags — both images failed to build. Use golang:1.23 and node:20-slim. (debian:trixie-slim IS valid.)
  • ZFS on trixie (resolved 2026-10-04): zfsutils-linux is NOT in trixie’s main repo — it lives in the distro’s own **contrib** component. There is NO external OpenZFS apt repo to add. Fix (in shared docker/deps.sh): enable contrib on the main+security suites, add a backports pin (src:zfs-linux → trixie-backports, priority 990) so apt picks the newer 2.4.4 over stable 2.3.9, then apt-get install zfsutils-linux. Only the CLI is installed — the kernel module is deliberately NOT built in-image (needs DKMS vs host kernel); privileged containers load the host’s ZFS module and just need the zfs/syncoid commands.
  • sanoid + syncoid are not packaged for Debian (verified: no apt package provides /usr/bin/syncoid). They live in jimsalterjrs/sanoid as single Perl scripts (no compile step; do NOT confuse with the C openzfs org or the unrelated simplesnap package). deps.sh installs pinned v2.3.0 into /usr/local/sbin plus runtime deps: perl, libcapture-tiny-perl (required by both), libconfig-inifiles-perl (sanoid config parser), pv/lzop/mbuffer (syncoid progress/compression, graceful if absent). Build fails fast if syncoid --version/sanoid --version don’t run.
  • Client image base: debian:trixie-slim + shared deps.sh (ZFS contrib + sanoid/syncoid + btrfs/rsync/restic) + openssh-client. Build with golang:1.23.
  • Server image: 3 stages — node:20-slim builds Next.js standalone, golang:1.23 builds the binary, debian:trixie-slim runtime copies .next/standalone → /app, .next/static → /app/.next/static, public → /app/public. Must mkdir -p /var/lib/weboid && chown weboid before USER weboid, else the state store can’t be created (permission denied).
  • Done (2026-10-04): serving the standalone bundle. Next.js standalone is an executable server.js, not static files. The Go server now picks a free loopback port, runs node .next/standalone/server.js with PORT=<free>/HOSTNAME=127.0.0.1, waits for it to accept TCP, and reverse-proxies / to it while /api/* stays on the Go mux (net/http/httputil.ReverseProxy). Verified end-to-end in Docker: /api/* 200 + all four UI routes 200.

Status / next steps

  • Repo skeleton, data model (Target/Job/Run), file-based state store
  • Runner interface with stubbed ZFS/btrfs/rsync impls
  • Outbound SSH transport (host-key pinning TODO) — note: original used ssh.DialContext which does not exist in golang.org/x/crypto/ssh; fixed to net.DialTimeout + ssh.NewClientConn so the module actually compiles
  • Server API (/api/health|targets|jobs|runs)
  • Both Dockerfiles + example compose files (base images fixed — see Docker notes)
  • MUI admin WebUI (Dashboard / Targets / Jobs / Runs) in src/ — was raw Bootstrap 5, then Tabler, now migrated to **@mui/material** v9 (AppRouterCacheProvider + dark theme). See the “weboid — MUI migration of the WebUI” session log for the v9 gotchas (separate icons package, renamed icon names, Grid2→Grid, reduced Stack API)
  • Serve Next.js standalone bundle properly: Go server launches node .next/standalone/server.js on a random loopback port and reverse-proxies / to it, keeping /api on its own mux
  • ZFS tooling in both images: zfsutils-linux from trixie contrib + pinned sanoid/syncoid v2.3.0 (Perl scripts from jimsalterjrs/sanoid). Verified: zfs 2.4.4, syncoid 2.3.0, sanoid 2.3.0 present; full Docker build + serving smoke test green
  • Client ↔ server over SSH + volume discovery (2026-10-05): client is now a real command dispatcher (weboid-client status → JSON of local ZFS/btrfs volumes); client image runs sshd so the server can dial in; server exposes GET /api/volumes (own host) + POST /api/targets/{id}/volumes (dials target over SSH). Shared internal/discovery package. Targets page has a per-target Discover button. Verified end-to-end against live ZFS on both a privileged client and the server, over real SSH (see session log for gotchas: zfs size property is invalid → use used/avail/refquota; sshd must be started by absolute path; capture stdout not combined output so logs don’t corrupt JSON).
  • ZFS backup path (2026-10-06): the first real transport works end-to-end. weboid-client backup-zfs <ds> <server-host> <remote-ds> lets syncoid snapshot locally and push the delta to the server’s pool; server exposes POST /api/targets/{id}/run and records a Run (running→succeeded/failed). Verified: 50MB file written to a source dataset appears byte-identical (md5 match) on the server’s receive pool; repeat runs transfer only the delta. The ZFS gotchas found while wiring this are in the session log + “ZFS/syncoid CLI gotchas” doc — the short version: syncoid 2.3.0 has no -s flag, can’t take a pre-made ds@snap as SOURCE (mis-parsed as user@host), refuses to receive into a pre-existing empty dataset, and ZFS 2.4’s zfs get <prop> <ds> parses <ds> as a property not a dataset.
  • Implement local half of the btrfs + rsync Runners (ZFS done)
  • Server-side receive for btrfs streams (syncoid receive is done via the server’s sshd)
  • Job scheduler (cron → trigger) + run history wiring
  • Per-target host-key pinning

Validation done

Rebuild notes (repo state 2026-10-05; repo dir deleted 2026-10-19)

  • weboid was a git repo at /data/code/weboid (Go module root, branch main, 51 files, repo-local identity Niklas Stephan <pooh85@gmx.de>, no remote configured; a push target would likely be github.com/niklasstephan/weboid per the go.mod path - confirm before pushing). .gitignore covered built binaries, /out/, .env*, Next.js artifacts.
  • WebUI build stage in docker/server.Dockerfile: Node 26 + pnpm. Gotchas: node:26-slim bundles no corepack/pnpm - install in-image (npm install -g corepack@latest && corepack enable, then corepack pnpm ..., COREPACK_ENABLE_DOWNLOAD_PROMPT=0); base pinned node:26.10.0-slim; the pnpm supply-chain policy (minimumReleaseAge + minimumReleaseAgeExclude) lives in src/pnpm-workspace.yaml, is load-bearing, and must be COPY’d before the install step (COPY src/package.json src/pnpm-lock.yaml src/pnpm-workspace.yaml ./) or the build fails on freshly-published deps. Client image is pure Go, unaffected.
  • WebUI was on-spec: Node 26 (.nvmrc=26, engines >=26), TypeScript 7.0.2, oxlint 1.87.0; lint/tsc/build all clean under Node 26.10.0.

WebUI framework decision (2026-10-07)

The user was unhappy with the raw Bootstrap/Tabler tables and moved the whole admin console from Tabler to MUI (**@mui/material** v9) — replacing @tabler/core (Bootstrap 5 under the hood) and @tabler/icons-react. This is a full rewrite of the UI layer, not just the tables.

  • Install (pnpm, store at /data/code/.pnpm-store): @mui/material, @mui/material-nextjs, @emotion/react, @emotion/styled, and **@mui/icons-material** (icons are a separate package in v9 — easy to miss).
  • Dark mode is the default look (matches the old data-bs-theme="dark"): createTheme({ palette: { mode: "dark" } }) + ThemeProvider + <CssBaseline />.
  • Full session log + v9 gotchas (renamed icon names, Grid2→Grid, reduced Stack API) in the child doc “weboid — MUI migration of the WebUI (2026-10-07)”.

MUI MCP server is enabled in this environment. When implementing or fixing MUI code, consult the MUI MCP tools for current component APIs/props rather than relying on training data — same “check official docs first” rule as other standard software (see AGENTS.md).

Source: Wiki.js