niklas‑stephan.de

Playbook for risky production changes (migrations, upgrades, cutovers)

Last updated 05/11/2026 · ops, process, docker, incident

Origin: Stalwart 1.0 attempt 2026-10-26 (see stalwart-1-0-upgrade): ~54 min crash loop + ~33 min second outage, logs lost, wrong root cause recorded. These rules would have prevented it.

Before touching anything

  1. Write the plan, end state and rollback command first (in the runbook file next to the compose project). Define abort criteria up front, e.g. “more than 3 container restarts or unhealthy for 2 min => revert immediately, no further forward work”.
  2. Record a baseline: docker ps / docker inspect (image digest, container id, StartedAt, health), a green end-to-end check (/data/maintenance/stalwart-cli/imap_check.py for mail), fresh Backrest backup.
  3. Rehearse on a copy: copy the store, start the new version with --network none (or a separate compose project/network, alternate ports, different container name). Prove it opens the data and passes the checks before production is involved.

While changing

  1. Never edit the production service block in place while preparing. Put helper services behind a compose profile, use separate paths. The production block changes only in the single, announced cutover step.
  2. Name the service in every compose command (docker compose up -d stalwart, never bare up -d). Run docker compose config and review the diff before applying.
  3. One writer per store: never run two containers against the same RocksDB/SQLite (LOCK error). Do not mix tool builds (e.g. migrate 1.0 vs 1.0-oss) on one state dir.
  4. After every production change, verify within 60 s (health, restart count docker inspect -f '{{.RestartCount}}', external check). A restart loop is a revert signal, not a debugging session on live mail.
  5. Capture evidence before it is destroyed: docker compose up/rm deletes the container and its logs. Before recreating, run docker logs --timestamps <c> > <project>/logs/<name>-<UTC>.log 2>&1 and docker inspect <c> > ...json.
  6. Timezones: write all times in the runbook/memory as UTC (or with explicit CET). Journald is local CET, migrate/docker logs are UTC; mixing them cost hours of misreading in the Stalwart incident.

Afterwards

  1. Do not record a root cause you have not reproduced. Label it “hypothesis” until a test reproduces it; keep the fact (what was observed) separate from the explanation. The Stalwart memory claimed an “edition flag” failure that a probe on a copy of the migrated store disproved.
  2. After tests/rehearsals, delete temp copies (large) and verify production health again.
  3. Update memory + runbook in the same session while the facts are fresh; commit via mem save.

Source: Wiki.js