niklas‑stephan.de

Stalwart 1.0 upgrade: migration procedure (not in-place)

Last updated 05/11/2026 · mail, handtrixxx.com, upgrade

Current state (2026-10-27 11:30 CET): PRODUCTION RUNS v1 (stalwartlabs/stalwart:v1, volumes data/etc-v1 + data/data-v1, env STALWART_PUBLIC_URL=https://mail.handtrixxx.com, TCP healthcheck kept). Cutover done 10:29-10:30 UTC, downtime ~67 s: fresh target + state dir (old ones kept as data-v1.old-2026-10-26, migrate-state.old-2026-10-26), prepare --non-interactive + copied answers.toml (all recommended answers; plain --non-interactive alone leaves questions unanswered), incremental run 25 s, stop 0.16, reflink snapshot data/data.pre-cutover-2026-10-27 (instant, cp -a --reflink=always), run --final --yes --owner 1000:1000, verify 8 accounts / 0 mismatches / 0 errors, docker compose up -d stalwart. Verified: restarts 0, healthy, public Traefik routes + TLS, IMAP login, JMAP, internal + outbound test mail, 12 DKIM keys (RSA+Ed25519) active, imapsync SUCCESS in the next cycle, stable 5+ min. Rollback: compose.yml.bak-pre-v1-cutover-2026-10-27 -> docker compose up -d stalwart returns to 0.16.27 on the untouched data/data (mail since cutover is only in v1; use migrate rollback into a new store for that). Delete the old stores/snapshot (~16 GB) only after v1 has run stably for some days. Known side effects to expect: OAuth clients re-login, DAV clients refetch, search index rebuild, 12 settings dropped (see migrate-state/final-report.txt). Container logs are empty via docker logs (logging goes elsewhere/none); use the API/stalwart-cli for diagnostics. Helper scripts: /data/maintenance/stalwart-cli/.

History of the failed 2026-10-26 attempt (superseded, kept for lessons): 0.16.27 was reverted then; cause was NOT a proven v1 blocker. Facts (UTC = CET-1): migrate prepare/run/run --final/verify all succeeded with tool image stalwartlabs/stalwart-migrate:1.0 (enterprise build, state.sqlite edition=enterprise), 0 errors, 26 warnings, 8 accounts 0 mismatches. A fresh probe on 2026-10-27 (v1 image, --network none, on a copy of data-v1) opened the store with no edition/schema error and bound the same 7 listeners as 0.16 (25,465,993,995,4190,443,8080). So the earlier memory claim (OSS flag 0x80000007, v1 refuses the store, un-flag key [b'u',0]) is disproved/unverified - do not act on it. Pairing that works: server stalwartlabs/stalwart:v1 + tool stalwart-migrate:1.0 (plain, not -oss); upstream ties the schema version to the edition (release notes), so tool build must match the server build. What really happened (journal CET): the stalwart container crash-looped (exit 1 after ~1 s, 57 restarts) 20:49-21:37, i.e. ~54 min mail down before the migrate runs (imapsync exit 102 19:49-20:42 UTC); cause unknown, logs lost when the container was recreated. After run --final (20:42:59 UTC) the container started 21:43:30 CET and ran stable until 22:16:29 CET (imapsync SUCCESS throughout), then was stopped; 22:15/22:16 re-run attempts failed (LOCK held by the running server; then final run already completed ... --force-after-final; then state DB created by the enterprise edition but this binary is community when the -oss tool was tried). Mail was down 22:16-22:49 CET; revert to pinned v0.16.27 at 22:49:50 CET. Why the cutover was abandoned at 22:16 is not recoverable from host logs. Lessons: see risky-change-playbook. Artifacts kept for a retry: migrated store ./data/data-v1, v1 config ./data/etc-v1, migrate state ./data/migrate-state/. Rehearsal PASSED 2026-10-27: v1 on a copy of data-v1 (--network none, STALWART_PUBLIC_URL set) was stable, health 200, WebUI/IMAP/SMTP/JMAP/OIDC-URL checks identical to prod, local agent->arcane delivery OK (helper /data/maintenance/stalwart-cli/rehearsal_check.py, needs a PROXY v1 header: our listeners trust loopback/Docker nets as PROXY sources and reset plain connections, prod too). Not covered: Traefik routing, outbound/DKIM, DAV, ACME/DNS, index rebuild time. The copy is frozen at 2026-10-26 20:43 UTC, so a real cutover needs a fresh run + run --final (0.16 stopped) first, then cutover per runbook Phase 5 with the abort rule from risky-change-playbook. No cutover is scheduled; needs Niklas’ go-ahead and a maintenance window.

Verified 2026-10-26 from official docs (stalw.art/docs/migration/upgrade-0-16/ and the v1.0.0 release notes): v1.0.0 has breaking changes and a new database layout — existing v0.16.x deployments cannot be upgraded in place. Migration uses the stalwart-migrate tool (separate repo: stalwartlabs/stalwart-migrate).

Plan written 2026-10-26, NOT executed: full runbook for this deployment at /data/docker/stalwart.handtrixxx.com/MIGRATION-1.0.md (phases: backup check -> migrate-profile service in compose.yml -> prepare -> incremental run while live -> final run in a window -> cutover to v1 with etc-v1/data-v1 volumes -> verify via imap_check.py/send_test.py; rollback = keep old store). Deployment-specific gotchas (verified 2026-10-26):

  • Stalwart 0.16 here is the Community/OSS build (SCIM 403, no license), BUT the correct migrate tool is the plain **stalwartlabs/stalwart-migrate:1.0** matching the published stalwartlabs/stalwart:v1 image (verified by probe 2026-10-27; the earlier advice to use -oss was wrong for the official v1 image). Docs: -oss is only for a server compiled without the enterprise feature. Never mix tool builds on one state dir.
  • v1 image needs for our setup (docs, untested): compose healthcheck override (image default curls 443/8080 and fails on our migrated PROXY-protocol listeners; or set STALWART_HEALTHCHECK_URL), STALWART_PUBLIC_URL=https://mail.handtrixxx.com behind Traefik (else sign-in Client metadata documents are disabled), outbound HTTPS to GitHub on first start (WebUI download), check Http.clientIpSource/PROXY trusted networks recorded by prepare. Image tag is v1 (there is no 1.0 image).
  • /etc/stalwart/config.json is only the store definition ({"@type":"RocksDb","path":"/var/lib/stalwart/data"}); every setting lives inside the RocksDB store. etc/config.toml is a legacy file that is not loaded - do not migrate it.
  • Certs are read from the Traefik cert dump (%{file:/traefik-certs/...}%), so the migrate container must mount ./traefik-certs too.
  • Containers run as uid 1000 while the migrate image is uid 2000, so every run needs --owner 1000:1000.

Requirements

  • Source must be exactly 0.16.27 — done on 2026-10-26: production is on 0.16.27 (verified docker exec stalwart stalwart --version, healthz 200, IMAP check OK), so the baseline for migration is already correct. Patch bumps are safe; docker compose up -d (not restart) applies a new image.
  • Free disk for a second full copy of the data store. Our store: RocksDB at /data/docker/stalwart.handtrixxx.com/data/data (~6.8 GB); /data has ~612 GB free — comfortable.
  • A maintenance window for the final copy (0.16 must be stopped).

Procedure (Docker)

  1. Add a migrate service to the compose file sharing Stalwart’s volumes/network/env, image stalwartlabs/stalwart-migrate (or the -fdb build for FoundationDB). Commands: docker compose run --rm migrate <cmd> ... — no sudo/--owner needed (same user as the Stalwart container).
  2. Create an empty 1.0 store + config, e.g. { "@type": "RocksDb", "path": "/var/lib/stalwart-v1/" } (new volume + config file).
  3. prepare --source <0.16 config.json> --state /var/lib/stalwart-migrate — converts settings, records decisions; review prepare-report.txt (dropped/changed objects).
  4. run (repeatable, incremental) while 0.16 keeps serving; poll with status.
  5. Stop 0.16, then run --final (confirms nothing changes for 30 s, finalises + verifies a sample).
  6. report --severity error (and optionally verify --full) before starting 1.0.
  7. Start stalwartlabs/stalwart:v1 with the new config + new data volume. Keep the 0.16 store and state dir until 1.0 proves itself; rollback can copy a 1.0 store back into an empty 0.16 store.

Known side effects of 1.0

  • Search index is rebuilt after start (incomplete results until done).
  • OAuth tokens are NOT migrated — users re-auth in OAuth clients.
  • DAV client ETags change.
  • Identifiers (account ids like d, g used in our JMAP scripts) are preserved. Docs also advise upgrading only when needed; 1.0 marks the start of automatic upgrades going forward.

Source: Wiki.js