CueCruxVaultCrux

Operations · 7. Daily operation

A Crux daemon fails slowly and predictably: disk fills, journals grow, an ingest valve trips, a queue stops draining. None of those are sudden. This chapter is a how-to: what to look at, how often, what normal looks like, and which numbers move before something breaks.

7.1 The five-minute morning check

In this order. If any step is not normal, stop and go to chapter 8.

1. Is it ready.

curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:14800/readyz

200. A 503 names the failing gate in its body, read it before doing anything else.

2. Open the console and read one widget. The status pill in the rail. You want Connected · Local or Connected · Platform. Degraded means the daemon answered but the console summary did not; Offline means it did not answer at all.

3. Rings › Activity, left column. The Needs you panel. This is the whole point of the console: everything that is waiting on a human is here, in three flavours, a gate awaiting approval, a plan that is blocked, and a session whose intent note suggests it is waiting for you. Clear the first, read the second, judge the third.

Remember the third is inferred, not reported, the coordination plane has no structured waiting-for-input field, so those cards carry an inferred from intent chip (render.js:2018).

4. Same screen, right column. The Fleet panel's four chips: coord, punchcards, sessions, orchestrators, each reading on, off or n/a. A chip that changed from on to n/a overnight is a feed that started erroring.

5. Disk.

df -h /path/to/data_dir

Below 15% free is your warning line. Below 10% the data_dir_capacity readiness gate takes the daemon out of rotation and the capacity guard pauses ingest on its own. §7.5 covers why that specific failure is so confusing when it happens.

7.2 What normal looks like, screen by screen

Knowing what normal looks like is most of the value of a console. Here is the reference.

ScreenNormalNot normal
Status pillConnected · …, and · read-only only if you expect customer postureDegraded persisting past a page reload
Rings ringdata-src="live" on the root once you have ≥50 qualifying ExecPlan itemsPlans you do not recognise; you are on the built-in snapshot
Rings › Live boardSessions you can account for, each with a recent "seen Ns ago"A session announcing a plan you retired
Rings › PunchcardsLeases whose holders are currentLeases held by a passport with no live session, a crashed agent
Work › ExecPlansBlocked column short and movingBlocked column growing week on week
Work › ActivityRows arriving; Go live shows new onesLive (reconnecting) persisting
Work › SessionsThe allocation panel's identity and plan-link shares steady or risingEither share falling, agents stopped binding
Memory › Factsshown of visible · stored all moving togetherstored climbing much faster than visible, churn, or a stateless client writing session bindings
Memory › ReviewLive contradictions near zeroA rising contradiction count on one entity
Trust › GatesEmpty, or short and movingA gate older than your review cadence
Trust › ReceiptsNew receipts appearingNone since a known write
System › Settings › Runtime capabilitiesEvery row you rely on availableAny row you rely on flipping to degraded

7.3 The weekly pass

Five things, none of which is urgent on any given day and all of which are urgent eventually.

1. Growth. Six artefacts have no automatic reclamation path at all: observation journals, shard quarantine directories, the routing shard-map history, the integrations audit journal, the activity journal and the cost report journal. Several other journals have no compaction equivalent to the fact journal's. Check their sizes against last week.

2. Fact-store churn. In Memory › Facts, compare visible against stored. A large and growing gap means retired facts accumulating. The single most common cause of unexpected growth is a stateless MCP bridge that re-initialises on every poll: each initialisation writes one durable session-binding fact, and without the ephemeral GC that population is unbounded.

The fix is CORECRUXD_EPHEMERAL_GC=1, which sweeps hourly, caps session bindings at 32 per passport and drops reverify receipts older than 30 days. It is read once at boot, so it needs a restart, and it never touches user facts (ephemeral_gc.rs:23).

3. The blocked column. Work › ExecPlans, Blocked. For each card, decide: is this waiting on a gate (go approve it), waiting on information (answer it), or dead (close it)? A blocked column nobody triages becomes a blocked column nobody reads.

4. Session allocation. Work › Sessions, the allocation panel. If the passport-binding or plan-link share is falling, your agents have stopped identifying themselves, and every downstream attribution, receipts, cost, gates, degrades with it. The daemon prints its own why string under the stats.

5. Tool surface. Rings › Agent, in Professional mode. Look at two columns: tools with a high error percentage (friction to fix) and tools marked never offered or ignored (candidates to demote from the surface). The 30-day window makes this a weekly-cadence read, not a daily one.

7.4 The monthly pass

TaskWhy
Verify a backup restoresA restore that has never been tested is a hope. Chapter 9 §9.4
Compact the fact journalOperator-triggered only. Rewrites the journal, dropping deleted values entirely
Reap quarantine and old observation journalsNothing else will
Review the trusted keyringStudio › Integrations › Trusted keys. Every signed manifest verifies against it
Re-read Runtime capabilitiesSystem › Settings, top card. Capability availability changes with configuration you may not have made
Check route-auth modeIf it is still shadow, decide whether the soak is over

7.5 The six signals that come before a failure

These are ordered by how often they actually precede an incident.

1. corecrux_data_dir_free_ratio falling below 0.15. The data_dir_capacity readiness gate flips at 0.10 by default, so 0.15 gives you warning before readiness does.

This is the single most confusing failure in the system, because of how it presents. A full data partition takes an otherwise-healthy daemon out of rotation, and downstream tooling reports it as a bare timeout with empty stderr, every integration test failing at once, across unrelated changes, with no useful message. df -h on the data partition is the first command, not the fifth.

2. corecrux_valve_pause_ingest at 1 when you did not pause it. Read CONTROL.json. If valves.pauseIngest.actor is capacity_guard, the background guard paused ingest on disk pressure and will resume at CORECRUXD_CAPACITY_RESUME_FREE_RATIO (config.rs:1112). It will not stomp an operator-set pause. Treat it as signal one with a different name.

3. corecrux_segment_corrupt_total increasing. Sets a flag that fails readiness gate seven until an operator clears it. One increment is a hardware or filesystem question, not an application one.

4. corecrux_log_redactions_total increasing while redaction is in audit mode. This is the pre-flight signal before switching redaction on: it tells you redactable material is passing through the logger and is currently only being counted (redaction.rs:26).

5. The Blocked column growing. Not a metric, a screen. Work is arriving faster than a human is unblocking it.

6. The Gates queue ageing. A gate older than your review cadence means an agent is stalled and nobody has noticed. The card shows its age in words for exactly this reason.

7.6 Five alerts that earn their place

AlertSignalWhy it earns its place
Disk approaching the gatecorecrux_data_dir_free_ratio below 0.15Warns before readiness does
Corruption detectedcorecrux_segment_corrupt_total increasingFails readiness until cleared
Ingest auto-pausedcorecrux_valve_pause_ingest at 1If the actor is capacity_guard, look at disk, not at your own actions
Redactable material in logscorecrux_log_redactions_total increasing in audit modeTells you what switching redaction on would remove
Unwitnessed headscrux_witness_unwitnessed_heads growingOnly meaningful when witnessing is enabled

Restrict /metrics at the network layer. It is unauthenticated, it is not covered by the minimal-probes flag, and its labels carry shard ids, node topology, valve state and hashed tenant ids.

7.7 The habits that make the console tell the truth

Four things you can do that cost nothing and make every later question answerable.

Bind a passport on every browser you use. Trust › Pending mints, Bound approver. Without it, every gated write refuses, and, more importantly; nothing you approve is attributable.

Ask your agents to set a title and a summary at save_session. Work › Sessions leads with the agent-given title when there is one; without it, the list is a column of ids. The empty row says so explicitly: no state summary, agents: set title/summary in save_session state.

Post cost reports. Both Meters pages are empty until corecruxctl session cost --post runs. Nothing computes them for you.

Set CRUX_EXECPLANS_ROOT in the daemon's process environment. Without it, the work board is only the kanban table, the ExecPlan projection has no files to read, and half the console's most useful joins have nothing to join to.

7.8 What the console will not tell you

Stated as plainly as the rest, because a routine built on a false assumption is worse than no routine.

  • The console does not alert. It has no notification, no sound, no badge count that follows you. Rings › Activity is a screen you have to look at.
  • A "may need input" card is inferred from free text. It is a prompt to look, not a report.
  • A coordination heartbeat is passport-level, not session-level. Sibling sessions of one identity share it, so "fresh" means that identity is around (render.js:1975).
  • A window-inferred cost attribution is not a link. It means those plans were open at the same time.
  • A receipt is a verifiable record of what was stored and retrieved. It is tamper-evidence, not attestation of what an agent did.
  • Three Explore surfaces are demo fixtures, Signals, Sourcing and Reverse. They carry a demo chip and have no endpoint behind them.

Sources