CueCruxVaultCrux

Operations · 10. Upgrade and rollback

Upgrading is: back up, stop cleanly, start the new binary against the same directory, verify. Rolling back is the same sequence in reverse, with one caveat this chapter states honestly rather than glossing. This how-to covers both.

10.1 The two update mechanisms, and what each actually does

They are different things and conflating them causes surprises.

MechanismWhat it isDefaultNetwork
crux self updateAn explicit, foreground operator command. Fetches the release manifest over HTTPS, compares versions, downloads the platform asset, verifies its sha256 against the manifest, and atomically replaces the running executable (self_update.rs:6)Never automatic, never a background defaultOnly when you type the command
The background update checkerCompares the local checkout against a git remote to populate the version posture on /v1/versionCORECRUXD_UPDATE_CHECK_ENABLED, on (config.rs:1313)Shells out to git on an interval

Three things about self update worth knowing before you rely on it:

  • --check reports without changing anything. Use it in a pre-flight.
  • A packaged install fails closed. If the install also carries crux-hook or corecruxctl, the command refuses and routes you to the complete installer or package manager, so companion versions cannot silently skew.
  • The trust ceiling is sha256 against an HTTPS-fetched manifest, not in-process signature verification. Full keyless verification is the install.sh path, and the command points you there when it cannot proceed. That is the honest boundary; it is stated in the source and it is stated here.

Neither mechanism replaces a deployment process. In a container, in Kubernetes, or behind any config-managed service unit, upgrade by replacing the image or the binary through your existing pipeline.

The background checker also has a container-specific failure mode: with no git checkout it produces noise, or worse, a confidently-wrong drift count. The shipped quickstart compose sets CORECRUXD_UPDATE_CHECK_ENABLED=0 for exactly that reason.

10.2 The safe upgrade sequence

1. Read the changelog and the known-defects chapter for the target version. A defect that is fixed changes behaviour you may be working around. Chapter 15 is the list.

2. Back up the data directory. Chapter 9. This is not optional on an upgrade; it is the only rollback you are guaranteed to have.

3. Note the running version and posture so you can tell whether step 6 worked:

curl -s http://127.0.0.1:14800/v1/version

4. Stop the old process cleanly. SIGTERM, not SIGKILL. The drain cap is CORECRUXD_SHUTDOWN_DRAIN_SECS, default 30 seconds; 0 drains without bound (config.rs:144). A hard kill during a journal append is the one write path where the fsync discipline cannot save you.

5. Start the new binary against the same data directory. First boot after an upgrade replays every JSONL journal and rescans .ccxi companions, so it is slower than a warm restart. If you have a startup probe, raise its initial delay for this boot or it will kill the process mid-replay.

6. Verify, in order.

curl -s -o /dev/null -w 'readyz %{http_code}\n' http://127.0.0.1:14800/readyz
curl -s http://127.0.0.1:14800/v1/version
curl -s 'http://127.0.0.1:14800/v1/facts/list?limit=1'

/readyz 200, then a version that is the one you deployed, then a substantive read whose total_nondeleted matches what you saw at step 3.

7. Open the console and check two cards. System › Settings › Runtime capabilities, and the status pill. A capability that was available before the upgrade and is now unavailable is the fastest signal that configuration and code have diverged.

10.3 What breaks across an upgrade

Ordered by how often each actually bites.

A container built without --build-arg GIT_SHA self-reports (unknown). The build context excludes .git, so the git fallback cannot fire. This makes "is the new binary actually running?" much harder to answer at exactly the moment you need to. Pass the argument.

A configuration variable that was a no-op may still be a no-op. CORECRUX_LOG_FORMAT is the standing example: every shipped manifest sets it and no code reads it. An upgrade does not fix a variable name.

A boolean that worked may stop working. Nine parsing rules exist across the flag surface and six flags do not trim whitespace. If your deployment tooling changed how it renders environment values, a flag that was 1 may now be 1\n. Use =1 and =0.

Flag defaults are the thing to diff. Read the new version's defaults against your explicit settings. A feature whose default flipped changes behaviour with no change on your side. The console tells you which are on: every gated pane names its flag in its empty state.

The console's service worker has its own revision. SW_REV is a constant that must be byte-identical between shell.html and sw.js, and is asserted by a test (sw.js:22). It changes with console releases. If the console looks stale after an upgrade, a hard reload or clearing the site data resolves it, the worker never caches /v1/*, so no data can be stale, only the shell.

Route-auth enforcement. If you are moving from shadow to enforce as part of the upgrade, do it as a separate change. Expect console routes to start returning 403 and soak it first. Two variables at once is two incidents.

10.4 Rolling back

The mechanical sequence is symmetrical: stop cleanly, put the old binary back, start against the same directory, verify with the same three commands.

The question that matters is whether the old binary can open the directory the new one has been writing to.

What is known

  • The data directory carries no explicit schema-version marker that the daemon checks at boot, and there is no migration runner in the daemon. Boot replays the journals and rescans companions; it does not consult a version stamp and refuse.
  • The journals are append-only JSON lines. A reader that encounters a field it does not know will not, in general, fail on that alone.
  • New on-disk artefact types are wired in three places, the storage allowlist, the projection registry, and load-at-startup. An artefact type introduced by a newer version and absent from an older one's allowlist is the shape of a quarantine-on-restart problem.

What is not known, and is not being asserted

Whether an older binary can safely open a data directory that a newer binary has written is not something this documentation can guarantee. There is no compatibility matrix in the repository, no schema-version gate to read, and no test in the tree that pins the behaviour. Anyone telling you "downgrades are fine" is extrapolating, and so would this page be.

Therefore: the rollback procedure that does not depend on the unknown

Roll back to the backup, not just to the binary.

  1. Stop the new daemon cleanly.
  2. Move the current data directory aside. Do not delete it; you may need to reconcile writes made since the upgrade.
  3. Restore the pre-upgrade backup (chapter 9 §9.6).
  4. Start the old binary against the restored directory.
  5. Verify: /readyz 200, /v1/version is the old version, a substantive read, a receipt verification.

You lose everything written between the upgrade and the rollback. That is the price of not having a compatibility guarantee, and it is a price you should assume when you plan the change window, not discover during it.

If you cannot accept that loss, do not upgrade a production daemon without first running the new version against a copy of the directory in a scratch environment, then attempting the downgrade there. That test takes ten minutes and converts an unknown into a measurement for your specific version pair.

10.5 Upgrading a container or a Helm release

ShapeSequence
ComposeBack up. docker compose down. Change the image tag. docker compose up -d. Verify
KubernetesBack up the PVC or its snapshot. Change the image. Let the rollout replace the pod. Raise the readiness probe's initial delay for the replay boot
Standalone binaryBack up. SIGTERM. Replace the binary. Start. Verify

Two shape-specific notes:

  • The container runs as UID 65532. A bind-mounted host directory must be owned by it. create_dir_all on an unwritable data directory is startup-fatal.
  • The shipped root compose sets restart: unless-stopped deliberately, so the console's Restart daemon button works: the process exits cleanly and the runtime brings it back. If your unit has no restart policy, that button stops the daemon and nothing restarts it.

10.6 The upgrade checklist

#StepSkip it and
1Read the changelog and known defects for the targetYou work around a defect that no longer exists
2Back up the data directoryYou have no rollback
3Record the running version and a fact countYou cannot tell whether the upgrade worked
4Stop with SIGTERMYou risk a torn journal append
5Raise the startup probe delayThe probe kills the process mid-replay
6Start with --build-arg GIT_SHA if building an imageThe binary reports (unknown)
7Verify /readyz, /v1/version, one substantive readYou find out from a user
8Check Runtime capabilities in the consoleA pane silently disappears and nobody knows why
9Diff flag defaults against your explicit settingsA default flip changes behaviour with no change on your side
10Do not change route-auth mode in the same windowTwo variables, two incidents

Sources