Crux Daemon · 17. Operations
Three deployment artefacts ship with the daemon: a Dockerfile, two compose files and a Helm chart. Only two of the three start on their defaults. This chapter is how-to: what each artefact does, what to probe, what to back up, how to upgrade, and how to diagnose the failures that actually happen.
Configuration values are in chapter 5; the startup refusals are in chapter 4 §4.3; the disk layout is in chapter 6.
17.1 Before you deploy anything
Four decisions, in this order. Getting them wrong is most of the trouble people have.
1. Pick an auth mode. There is no default and the daemon will not start without one. For anything reachable, that means jwt_hs256 or jwt_jwks plus the corresponding secret. off and dev_scopes cannot bind a non-loopback address without an explicit override, by design.
2. Set CORECRUXD_DATA_DIR to an absolute path. The default is the relative ../CoreCruxData/v1, which resolves against the working directory. Two starts from two directories give you two silently divergent data dirs.
3. Decide what is reachable. The daemon has no TLS. Put a terminating proxy in front, or keep it on loopback and publish through the host. Restrict /metrics at the network layer; it is unauthenticated and exposes shard ids, node topology and hashed tenant ids.
4. Set LOG_FORMAT=json if you have a log pipeline. Not CORECRUX_LOG_FORMAT, which every shipped manifest sets and no code reads. See chapter 16 B1.
A minimal working environment for a networked deployment:
CORECRUXD_AUTH_MODE=jwt_hs256
CORECRUXD_JWT_HS256_SECRET=<at least 32 bytes>
CORECRUXD_DATA_DIR=/data
CORECRUXD_HTTP_HOST=0.0.0.0
LOG_FORMAT=json
CORECRUXD_REDACT=on
CORECRUXD_PASSPORT_CLAIM_ON_STARTUP=0The last two are deliberate: redaction defaults to counting rather than removing, and the passport claim is the one outbound call in a default configuration.
17.2 The container image
Dockerfile is a two-stage build.
| Property | Value |
|---|---|
| Builder | Upstream rust:1.88-bookworm, not a distroless base. The in-file comment (Dockerfile:5-9) justifies this as a deliberate exception: the workspace pins its toolchain through rust-toolchain.toml and needs rustup to honour that pin |
| Runtime | cgr.dev/chainguard/wolfi-base:latest (Dockerfile:42) |
| Runtime packages | ca-certificates, curl, git (Dockerfile:53). TLS for opt-in outbound features; curl for the healthcheck; git because the update checker shells out to git fetch against a /repo bind mount, without it the fetch silently fails and the banner reports a confidently-wrong drift count |
| User | USER 65532:65532, non-root. /data is chowned at build (Dockerfile:60-62) |
| Image env | CORECRUXD_DATA_DIR=/data, CORECRUXD_BUILD_CCXI=1, CORECRUX_LOG_FORMAT=json, the last is a no-op (Dockerfile:64-66) |
| Exposed port | EXPOSE 14800 only (Dockerfile:68). MCP on 14801 and gRPC on 4007 are not exposed by the image |
| Healthcheck | curl -f http://localhost:14800/readyz, 10s interval, 5s timeout, 3 retries (Dockerfile:70-71) |
| Volume | VOLUME ["/data"] (Dockerfile:73) |
| Binaries | Both corecruxd and corecruxctl (Dockerfile:55-56) |
| Features | None. Built with default features, so no OTel, no WASM host, no fastembed, no hosted surfaces (Dockerfile:36) |
Two consequences worth planning for:
- A bind-mounted host directory must be chowned by you. The container runs as UID 65532 and
create_dir_allon an unwritable data dir is startup-fatal. - A build without
--build-arg GIT_SHA=...self-reports(unknown). The build context excludes.git, so the git fallback cannot fire andcorecruxd --versionloses its commit.
17.3 The two compose files
docker-compose.yml at the repository root and examples/quickstart/docker-compose.yml differ in useful ways.
| Fact | Root compose | Quickstart compose |
|---|---|---|
| Auth mode | dev_scopes with CORECRUXD_ALLOW_INSECURE_DEV_AUTH_BIND=1 | the same |
| Published ports | 127.0.0.1:14800:14800 and 127.0.0.1:14801:14801, loopback-bound on the host. gRPC 4007 is not published | see the file |
restart | unless-stopped, explicitly so the console's "Restart daemon" button works: the process exits cleanly and Docker brings it back (docker-compose.yml:8-11) | - |
| Memory limit | 4G | 4G |
| Optional Ollama sidecar | --profile embeddings, pulls the embedding model on first start (docker-compose.yml:41-53) | not present |
CORECRUXD_EMBEDDING_URL and _MODEL | passed through from host env | not set |
CORECRUXD_OBS_RETENTION_DAYS | passed through; unset means retain forever | not set |
CORECRUXD_UPDATE_CHECK_ENABLED | not set | 0, "the container has no git checkout; disable the repo-drift probe so update posture is explicit, not unavailable" (examples/quickstart/docker-compose.yml:51-53) |
The root compose's 127.0.0.1:14800:14800 mapping is the pattern to copy. The container binds 0.0.0.0 inside its own namespace, and the host publishes only on loopback. You get the container's convenience without exposing an unauthenticated daemon.
17.4 The Helm chart
helm/corecrux/ ships Chart.yaml, values.yaml, a README and six templates.
| Fact | Value |
|---|---|
| Container port | 14800 only, named http (deployment.yaml:33-36) |
| Service | Single port, targetPort: http (service.yaml:8-13) |
CORECRUXD_DATA_DIR | /data, hard-coded |
CORECRUXD_HTTP_HOST | 0.0.0.0, hard-coded (deployment.yaml:40-41) |
CORECRUXD_AUTH_MODE | .Values.auth.mode, defaulting to off (values.yaml:34-35) |
CORECRUX_LOG_FORMAT | .Values.config.logFormat, a no-op |
CORECRUXD_BUILD_CCXI | .Values.config.buildCcxi |
| Extra environment | range .Values.env (deployment.yaml:48-52) |
| Liveness probe | GET /healthz, initial delay 5s, period 10s |
| Readiness probe | GET /readyz, initial delay 5s, period 5s |
A stock helm install with no value overrides CrashLoopBackOffs. HTTP_HOST=0.0.0.0 plus auth.mode: "off" trips the bind-posture rail, and the chart supplies no override. This is defect B2 in chapter 16, with the full diagnosis and the two workarounds.
Two further limitations to plan around:
- MCP and gRPC are unreachable in a default install. Only 14800 is a container port and only 14800 is on the service. If you need the MCP plane, add the port to both.
CORECRUXD_HTTP_HOSTis the only bind the chart sets. gRPC and MCP fall back to127.0.0.1, which is why the bind rail fails on the HTTP leg alone.
The chart also ships servicemonitor.yaml and prometheusrule.yaml, so a Prometheus Operator stack can scrape /metrics without extra wiring, but see §17.5 on why that endpoint needs a network-layer restriction.
17.5 Health, readiness and what to alert on
Probe with /readyz, not /healthz. /healthz always returns 200 and ok: true; nothing in the handler can make it fail. It is a liveness signal only. /readyz evaluates nine gates and returns 503 with the failing ones named. Both are unauthenticated. Full detail is in chapter 9 §9.6.
Probe configuration that works:
| Probe | Endpoint | Notes |
|---|---|---|
| Liveness | GET /healthz | Only detects a wedged or dead process |
| Readiness | GET /readyz | 200 means all nine gates pass; 503 lists the failures in checks[] |
| Startup | GET /readyz | There is no /startupz. Allow generous initial delay, boot replays every JSONL journal |
CORECRUXD_PUBLIC_PROBES_MINIMAL=1 strips the per-gate breakdown from a /readyz failure and the routing and valve detail from /healthz. It does not cover /metrics.
Five alerts that earn their place:
| Alert | Signal | Why |
|---|---|---|
| Disk approaching the readiness gate | corecrux_data_dir_free_ratio below 0.15 | The data_dir_capacity gate flips at 0.10 by default. This warns you before readiness does |
| Corruption detected | corecrux_segment_corrupt_total increasing | Sets a flag that fails readiness gate 7 until an operator clears it |
| Ingest auto-paused | corecrux_valve_pause_ingest at 1 | If the actor is capacity_guard, look at the disk, not at your own actions |
| Redactable material in logs | corecrux_log_redactions_total increasing while CORECRUXD_REDACT=audit | The pre-flight signal before switching redaction on |
| Unwitnessed heads accumulating | crux_witness_unwitnessed_heads growing | Only meaningful when witnessing is enabled |
Restrict /metrics at the network layer. It is classified Public with an empty scope set and is not covered by the minimal-probes flag. Its labels carry shard ids, node topology, valve state and hashed tenant ids.
17.6 Troubleshooting
Ordered by how often each actually happens.
The daemon exits immediately with a message and no log lines. You are in boot steps 1 to 9, before init_tracing. The message on stderr is the whole diagnosis. Match it against the 25 refusal conditions in chapter 4 §4.3, each row has the exact fix.
CORECRUXD_AUTH_MODE must be set explicitly. There is no default. Set the variable, or set daemon.auth_mode in the config file. If you thought you set it in a config file, see the next entry.
Configuration in a YAML file appears to be ignored. Three possibilities, and all three are silent. XDG_CONFIG_HOME is unset, so no file is read at all; there is no ~/.config fallback. Or the file has a YAML syntax error, and the parse failure is discarded. Or the path is wrong, which looks identical to no file. Set CORECRUXD_CONFIG_PATH explicitly and validate the YAML separately.
A flag on the command line does nothing. corecruxd accepts no runtime configuration flags and silently ignores unrecognised ones. corecruxd --data-dir /x starts normally and ignores the flag.
auth mode Off may not bind to non-loopback addresses…. The bind-posture rail. Switch to a JWT mode, keep the bind on loopback, or set CORECRUXD_ALLOW_INSECURE_DEV_AUTH_BIND=1 knowingly. On Helm this is the stock-install failure, see chapter 16 B2.
A boolean flag you set has no effect. Nine incompatible parsing rules, and six flags do not trim. A trailing newline from EnvironmentFile= or configMapKeyRef silently means off. Use =1 and =0, which work under every rule. See chapter 5 §5.5 and §5.6.
Everything unrelated is failing, /readyz returns 503, and the error is unhelpful. Check df -h on the data partition first. The data_dir_capacity gate takes an otherwise-healthy daemon out of rotation below 10% free, and downstream tooling reports it as a bare timeout with empty stderr. Reap shards/*/quarantine/ and observations/*.jsonl, neither has any GC path.
Ingest is paused and you did not pause it. Read CONTROL.json. If valves.pauseIngest.actor is capacity_guard, the background guard paused it on disk pressure and will resume at CORECRUXD_CAPACITY_RESUME_FREE_RATIO. It will not stomp an operator-set pause.
Logs are not JSON despite setting CORECRUX_LOG_FORMAT=json. No code reads that name. Set LOG_FORMAT=json.
Log level changes have no effect. RUST_LOG overrides CORECRUXD_LOG_LEVEL entirely when set.
An MCP client cannot connect after following the session handshake. channels.mcp from POST /session points at port 14800, where no /mcp route exists. Use http://127.0.0.1:14801/mcp. See chapter 16 B3.
A gRPC call returns unimplemented. Every registered RPC does, in every build from this repository. Use HTTP. See chapter 16 B5.
POST /v1/admin/append returns a platform-upgrade response. It always does. It is not a subscription problem. Use the fact and local-ingest surfaces.
A route you expected to be blocked is not. Route-auth defaults to shadow mode: it logs route_auth_shadow_mismatch and continues. Set CORECRUXD_ROUTE_AUTH=enforce, and expect the console routes to start returning 403 when you do.
The daemon starts but reports a different identity, and receipts stop verifying. passport.key is missing and a new one was generated silently. It cannot be recovered. It also encrypts stored integration credentials through a derived subkey, so those are gone too.
A second daemon started against the same data. The LOCK flock is per resolved path. Two starts from different working directories against the default relative data_dir do not collide; they diverge. Always set an absolute path.
A repo scan is marked failed with "daemon restarted before scan completed". Expected restart recovery, not corruption. Re-run the scan.
17.7 Backup and restore
Back up the whole data directory. It is the entire state of the daemon. Everything in chapter 6 §6.5 is in there.
If you cannot take the whole thing, the minimum viable set is facts.jsonl, passport.key, passports/, CONTROL.json, meta/, shards/, audit-export-signing.key, integrations/ and .install-uuid.
Four rules for taking a copy:
- Stop the daemon, or accept a fuzzy snapshot. There is no quiesce command. The journals are append-only and fsynced, and the segment writes are atomic, so a running copy is usually consistent, but "usually" is not a backup policy for the fact journal.
- Never back up only
facts.jsonl. Withoutpassport.key, receipts written by the old identity will not verify against the restored daemon. - Filesystem snapshots are the good answer where you have them. The layout is designed for it: temp-file-and-rename everywhere, fsync before rename, and a MANIFEST that is the authority for which segments are live.
- Test the restore. A restored data dir that starts and reaches
/readyz200 is a verified backup; anything less is a hope.
What restore does not recover:
- Device-authorization grants, process-local, invalidated by any restart.
- In-memory session state if
CORECRUXD_FACT_PERSISTENCEwas off. - The
LOCK, which is regenerated and should not be copied while a daemon is running.
17.8 Upgrade
The daemon has a self-updater (corecruxd self update, and self update --check), and a background update checker that compares against a git remote for the /v1/version posture. Neither replaces a deployment process.
A safe sequence:
- Read the changelog and this set's chapter 16 for the target version. A defect that is fixed changes behaviour you may be working around.
- Back up the data directory. See §17.7.
- Stop the old process cleanly, with SIGTERM rather than SIGKILL. The drain cap is
CORECRUXD_SHUTDOWN_DRAIN_SECS, default 30 seconds;0drains forever. A hard kill during a journal append is the one write path where the fsync discipline cannot help you. - Start the new binary against the same data directory. Boot replays every JSONL journal and rescans
.ccxicompanions, so first start after an upgrade is slower than a warm restart. Set your startup probe's initial delay accordingly. - Verify:
curl /readyzfor a 200, thenGET /v1/versionto confirm the version and feature posture, then one substantive endpoint.
Two upgrade-specific cautions:
- A container built without
--build-arg GIT_SHAreports(unknown), which makes "is the new binary actually running?" harder to answer. Pass the argument. CORECRUXD_UPDATE_CHECK_ENABLEDdefaults to on and shells out to git. In a container with no checkout it produces noise; the quickstart compose sets it to0for exactly this reason.
17.9 Capacity and growth
Six artefacts have no automatic reclamation path at all: observations/*.jsonl, shards/*/quarantine/, meta/routing/shardmap.v*.json, integrations/audit.jsonl, activity/journal.jsonl and cost/reports.jsonl. Several journals, witness_proofs.jsonl, entities.jsonl, substrate-edges.jsonl, sessions.jsonl, cases.jsonl, relations.jsonl, sync-outbox.jsonl, have no compaction equivalent to the fact journal.
Four things you can actually do:
| Action | Effect |
|---|---|
Set CORECRUXD_OBS_RETENTION_DAYS | Archives observation sessions hourly. Unset means retain forever |
Set CORECRUXD_EPHEMERAL_GC=1 | Hourly sweep of two reserved fact prefixes: session bindings capped at 32 per passport, and reverify receipts older than 30 days. Read once at boot, so it needs a restart. It never touches user facts |
| Compact the fact journal | Operator-triggered only. Rewrites facts.jsonl, dropping deleted values entirely |
Reap quarantine/ and old observations/*.jsonl | Manually. Nothing else will |
A stateless MCP bridge that re-initialises per poll writes one durable session-binding fact each time. Without the ephemeral GC that population is unbounded. This is the most common cause of unexpected facts.jsonl growth.
17.10 The operator CLI
corecruxctl ships in the same container image and is both a library and a binary. Two entry points are documented in-code as the canonical on-ramps:
corecruxctl start(main.rs:63-77), "START HERE, the one command to get live: detect daemon, authenticate, wire MCP + hooks, round-trip a first fact, print a 'you're live' summary." Flags:--url,--token.corecruxctl deploy-audit(main.rs:38-61), "Audit the target daemon's bind/auth posture before a networked deployment." Its--configdefault matches the daemon's own config discovery, so it audits the file the daemon would read. Run this before exposing a daemon.
Other operationally relevant handlers among its 40 modules: verify_store, replay, receipts, projections, shardmap, storage, snapshot, audit_export, evidence, incident, reconcile, smoke, code_health, output_verify.
Two cosmetic surprises: the CLI's about-string is still "CoreCrux v3 control tool (Phase 0)" against workspace version 0.5.52, and corecruxctl ops append is documented in a 444-line module that is wired to no subcommand.
17.11 Running it safely: a checklist
Ten items. Every one is grounded in a default that is not what a production operator would choose.
| # | Check | Why |
|---|---|---|
| 1 | CORECRUXD_AUTH_MODE is a JWT mode, not off or dev_scopes | The dev modes are unauthenticated self-assertion with no tenant isolation |
| 2 | CORECRUXD_DATA_DIR is absolute | The default is relative to the working directory |
| 3 | A TLS-terminating proxy is in front | The daemon has no TLS on any plane |
| 4 | /metrics is restricted at the network layer | Unauthenticated, and it leaks topology |
| 5 | CORECRUXD_REDACT=on if logs leave the host | The default counts without removing |
| 6 | LOG_FORMAT=json if you parse logs | The documented variable name is a no-op |
| 7 | CORECRUXD_ROUTE_AUTH=enforce, after a shadow-mode soak | The default logs violations and lets them through |
| 8 | CORECRUXD_PASSPORT_CLAIM_ON_STARTUP=0 for air-gapped or privacy-sensitive deployments | It is on by default and reaches the public internet once |
| 9 | CRUX_PASSPORT_REVOCATION is unset or exactly 1 | Any other set value silently disables it |
| 10 | Backups include passport.key, and a restore has been tested | Losing it loses receipt verification and every stored integration credential |
Sources
- Dockerfile, the container image
- docker-compose.yml, the root compose file
- examples/quickstart/docker-compose.yml, the quickstart compose file
- helm/corecrux/templates/deployment.yaml, the chart deployment template
- helm/corecrux/values.yaml, the chart defaults
- crates/corecruxd/src/http/health.rs:144, the
/readyzhandler - crates/corecruxd/src/main.rs:2168, the shutdown path
- crates/corecruxd/src/main.rs:2506, the capacity guard's autonomous valve write
- crates/corecruxctl/src/main.rs:38,
corecruxctl deploy-audit