CueCruxVaultCrux

Operations · 9. Backup and recovery

Back up the whole data directory. If you can only take one file, take passport.key, and understand that a backup without it is not a backup. This chapter is a how-to.

The disk layout itself is chapter 6 of the internals set. This chapter is what to do about it.

9.1 The one-sentence policy

The daemon's entire state is one directory. Back that directory up and you have backed up the daemon. Delete facts.jsonl from it and the daemon starts clean, healthy and empty, and will not tell you anything is missing.

That last clause is why this chapter is not optional. There is no alarm for "your memory is gone". /readyz returns 200 on an empty store.

9.2 What to copy

Take the whole directory. It is designed for that: temp-file-and-rename everywhere, fsync before rename, and a MANIFEST that is the authority for which segments are live.

If a constraint forces a partial copy, the minimum viable set is:

PathWhy it is in the minimum set
facts.jsonlThe memory itself
passport.keyThe daemon's identity. See §9.3
passports/The passport records
CONTROL.jsonValve and control state
meta/Routing and shard-map metadata
shards/The event store
audit-export-signing.keyAudit export signing
integrations/Installed integration state
.install-uuidInstall identity

Do not copy the LOCK while a daemon is running. It is regenerated on start.

9.3 Why passport.key is the file that matters

passport.key does two jobs, and losing it breaks both irreversibly:

  1. It is the daemon's signing identity. Receipts written under the old key will not verify against a restored daemon that generated a new one.
  2. It encrypts stored integration credentials through a derived subkey. Lose the key and every stored connector credential is unrecoverable ciphertext.

There is no recovery path. If the file is absent at boot, the daemon silently generates a new one and carries on. The first symptom you will see is receipts that stop verifying and an identity you do not recognise (chapter 8 §8.5).

Back it up separately, before anything writes. It is small, it is static, and it is the only file in the system whose loss cannot be worked around.

9.4 Taking a copy that is actually consistent

Four rules.

1. Stop the daemon, or accept a fuzzy snapshot. There is no quiesce command. The journals are append-only and fsynced, and segment writes are atomic, so a running copy is usually consistent, but "usually" is not a backup policy for the fact journal.

2. Filesystem snapshots are the good answer where you have them. The layout was designed for snapshot semantics; take one and copy from it.

3. Never back up only facts.jsonl. Without passport.key, receipts written by the old identity will not verify against the restored daemon. A journal-only backup restores the data and destroys the proof.

4. Order matters if you cannot stop the process. Copy passport.key and the small metadata files first, then the journals, then the shards. A journal that is ahead of its metadata replays; metadata ahead of its journal does not.

9.5 Proving a backup restores

A backup you have not restored is a hope. The test is small enough to automate and there is no excuse for skipping it.

# 1. Restore the copy to a scratch path.
cp -a /backups/corecrux-2026-07-27 /srv/restore-test

# 2. Start a daemon against it, on a spare port, loopback only.
CORECRUXD_DATA_DIR=/srv/restore-test \
CORECRUXD_AUTH_MODE=dev_scopes \
CORECRUXD_HTTP_PORT=24800 \
CORECRUXD_MCP_PORT=24801 \
CORECRUXD_GRPC_PORT=24007 \
  corecruxd &

# 3. The gate: readiness, then one substantive read.
curl -s -o /dev/null -w 'readyz %{http_code}\n' http://127.0.0.1:24800/readyz
curl -s http://127.0.0.1:24800/v1/version
curl -s 'http://127.0.0.1:24800/v1/facts/list?limit=1'

The pass criteria, in order:

CheckPass means
/readyz returns 200All nine gates pass on the restored directory
/v1/version returns the expected buildYou restored what you think you restored
/v1/facts/list?limit=1 reports a total_nondeleted you recogniseThe journal replayed and the store is populated
A receipt verifiesThe identity survived. This is the check that catches a missing passport.key

That last one is the whole point of the exercise. Open the restored daemon's console at Trust › Receipts, find a fetchable ad_ga_* row, expand it, and read the verdict. signature_valid: true with error_code: OK means the identity restored. Anything else means it did not.

Use different ports so the test never contends with production, and never point the test at the production directory; it would take the lock.

9.6 The restore procedure

1. Stop the daemon cleanly. SIGTERM, not SIGKILL. The drain cap is CORECRUXD_SHUTDOWN_DRAIN_SECS, default 30 seconds; 0 drains without bound (config.rs:144). A hard kill during a journal append is the one write path where the fsync discipline cannot help you.

2. Move the current directory aside rather than deleting it. You may need to reconcile.

3. Restore the copy in place.

4. Fix ownership. In a container the daemon runs as UID 65532 and create_dir_all on an unwritable data directory is startup-fatal. chown -R 65532:65532 the restored tree if that is your deployment shape.

5. Do not copy in a LOCK. If one came with the backup, delete it.

6. Start the daemon. The first boot after a restore replays every JSONL journal and rescans .ccxi companions, so it is slower than a warm restart. Set your startup probe's initial delay accordingly.

7. Verify, in this order: /readyz 200, then /v1/version, then one substantive endpoint, then a receipt verification.

9.7 What a restore does not bring back

Four things, stated as plainly as the rest:

Not restoredWhy
Device-authorization grantsProcess-local, invalidated by any restart
In-memory session stateIf CORECRUXD_FACT_PERSISTENCE was off, session state was never on disk
The LOCKRegenerated, and should not be copied while a daemon is running
Anything written after the copyObviously: but worth stating, because the daemon will not tell you there is a gap

There is also no partial restore. The directory is one unit; restoring facts.jsonl from Tuesday into a Thursday directory produces a store whose journal and metadata disagree.

9.8 A backup schedule that is defensible

CadenceAction
Once, before anything writesCopy passport.key to a separate, durable location
DailyFilesystem snapshot, or a stop-copy-start if you can take the downtime
WeeklyRetain one daily as the weekly
MonthlyRun §9.5 against the most recent backup. Not a spot check, the full four-step gate
On every upgradeTake a backup first. Chapter 10 §10.2

The monthly restore test is the item people drop, and it is the only item that distinguishes a backup from a directory of files.

9.9 What backup does not protect you from

Schneier's question, answered: what does this not prove.

  • It does not protect against a bad write. The fact journal is append-only; a wrong fact written yesterday is in every backup taken since. Use the as of control in Memory › Facts to find when a value changed, and consolidation to supersede it.
  • It does not prove the data was not tampered with. Receipts do that, and only for what they cover, a CROWN receipt is a verifiable record of what was stored and retrieved, which is tamper-evidence, not attestation of conduct.
  • It does not survive losing the key alone. A complete directory backup with a separately-lost passport.key restores the data and loses the proof. That is why the key gets its own copy, in its own place.
  • It says nothing about the hosted tier. If you rely on a mediated engine, its state is not in your data directory and this procedure does not cover it.

Sources