Crux Daemon · 9. Observability

Three probe endpoints, 142 Prometheus metrics, and nine readiness gates that must all pass for a 200 on /readyz. The gate that bites in practice is data_dir_capacity at 10% free, §9.6 explains why it makes unrelated things fail with an unhelpful message.

This chapter is reference.

9.0 In plain English

Observability is the set of ways the daemon tells you how it is doing without you having to ask it a question about your data. It comes in three forms here. Logs are what it says as things happen. Metrics are counters and gauges scraped by a monitoring system, 142 of them, exposed at /metrics in Prometheus format. Probes are the two endpoints a load balancer or an orchestrator calls to decide whether this process should receive traffic.

The two probes are not interchangeable, and confusing them is the most common mistake made here. /healthz is liveness: it answers "is this process alive and responding", it always returns 200, and nothing in the handler can make it fail. It is useless as a readiness signal because it cannot say no. /readyz is readiness: it runs nine independent gates and returns 503 with a per-gate breakdown if any one of them fails. If you have wired an orchestrator to /healthz expecting it to catch a sick instance, it will never catch anything.

You will touch this chapter when you first wire the daemon into a monitoring system, and then again the first time something goes wrong in a way the error message does not explain. That second visit is what §9.6 is really for. The gate that bites in practice is data_dir_capacity, which fails when the data partition drops below 10% free. When it trips, the daemon starts returning 503 on readiness, and everything downstream fails in ways that have nothing to do with disk: tests time out waiting for a healthy daemon, clients see connection failures, and the messages they print name none of it. The habit worth building is to check free disk before you believe any other theory.

The one thing people get wrong beyond the two probes: they assume a green start means logs are complete. Nothing is logged before step 10 of the boot sequence, which means the entire config parse and every auth-posture decision happen in silence. If a flag you set had no effect, no log line will tell you so, because the code that read it ran before logging existed. §9.8 gathers this and the rest of what observability here does not give you, including the fact that /metrics is unauthenticated and does expose shard ids, node topology, valve state and hashed tenant ids to anyone who can reach the port.

9.1 Logging

Setup is init_tracing (main.rs:2065), called from main at main.rs:421, step 10 of the boot sequence. Nothing is logged before that point, which includes the entire config parse and every auth-posture rail.

Level

// crates/corecruxd/src/main.rs:2066
let filter = tracing_subscriber::EnvFilter::try_from_default_env()
    .unwrap_or_else(|_| tracing_subscriber::EnvFilter::new(level));

Knob	Env var	Default	Notes
Filter directives	RUST_LOG	unset	try_from_default_env() reads it. Full EnvFilter syntax, e.g. corecruxd=debug,tower_http=warn. It wins outright over CORECRUXD_LOG_LEVEL
Fallback level	CORECRUXD_LOG_LEVEL	info	Consulted only when RUST_LOG is unset or unparseable (config.rs:843)

Format

// crates/corecruxd/src/main.rs:2068
let log_format = std::env::var("LOG_FORMAT").unwrap_or_default();

json, case-insensitively, selects the JSON layer (main.rs:2129). Anything else gives human-readable text.

The variable is LOG_FORMAT, unprefixed. CORECRUX_LOG_FORMAT is read by no code anywhere, yet it is what the Dockerfile, the Helm chart, both compose files, config.example.env and the quickstart README all set or document. JSON logging is silently off in every shipped manifest. See chapter 16 defect B1.

Sink-boundary redaction

Every formatted event is scrubbed before it reaches stdout, through a RedactMakeWriter wrapping std::io::stdout (main.rs:2072).

Knob	Values	Default
CORECRUXD_REDACT	on/1/true/enforce redacts; off/0/false/disabled passes through; audit counts hits without mutating. Unknown values fall back to audit	audit
CORECRUXD_REDACT_EXTRA_PATTERNS	;;-separated id=regex, e.g. myco=MYCO-[0-9]{6};;legacy=LK_[a-f0-9]{32}. Invalid entries are warn-logged and dropped	unset

The default counts but does not redact. Out of the box the daemon tells you how much redactable material is in its logs and ships it anyway. If you send logs off-box, set CORECRUXD_REDACT=on. The counter corecrux_log_redactions_total{rule} increments in both modes, which makes audit the correct pre-flight before switching to on.

The redactor is a process singleton published globally so other crates in the process scrub with the same instance (redaction.rs:41).

/v1/observe/* has its own, stricter knob: CORECRUXD_OBSERVE_REDACT defaults to on (observe_audit.rs:241).

Panics

A std::panic::set_hook emits a structured tracing::error! carrying panic.payload and panic.location (main.rs:457). Axum handler panics are caught separately and returned as an application/problem+json 500, with tracing::error!(panic = %msg, "handler panicked") (health.rs:365).

The structured operations log

structured_log.rs defines a one-JSON-line-per-operation record. Fields (structured_log.rs:127): ts (RFC3339 with milliseconds, UTC), level, request_id?, trace_id?, traceparent?, op, outcome, took_ms, shard_id?, epoch?, error_code?, retryable?, retry_after_ms?, error_detail?, payload_hash_prefix?. Optionals are omitted when absent.

• error_code uses the log taxonomy, not the HTTP one. See chapter 8 §8.6.
• Correlation ids: x-request-id and traceparent are lifted from HTTP headers or gRPC metadata and sanitised (structured_log.rs:49), no ASCII control characters, non-empty, at most 128 characters, restricted to [A-Za-z0-9] plus - _ . : / =. A traceparent additionally needs at least 16 characters. A missing or rejected x-request-id is replaced with a fresh UUIDv4.
• payload_hash_prefix is the first 12 hex characters of BLAKE3 over the payload (structured_log.rs:124). Payloads themselves are never logged.

9.2 The metrics endpoint

Property	Value
Path	GET /metrics (http/mod.rs:475)
Content-Type	text/plain; version=0.0.4; charset=utf-8 (health.rs:352)
Encoder	prometheus::TextEncoder over registry.gather() (metrics.rs:2177)
Auth	None. Classified Public with an empty scope set (route_auth.rs:79)
On failure	500 application/problem+json

/metrics is unauthenticated and is not covered by CORECRUXD_PUBLIC_PROBES_MINIMAL; that flag trims only /healthz and /readyz. Metric labels expose shard ids, node topology, valve state and hashed tenant ids. Restrict /metrics at the network layer if the daemon is reachable beyond loopback.

A single prometheus::Registry is created at metrics.rs:159 and shared, so every subsystem registers against the same scrape.

Total: 142 metrics, across four registration sites, all of which run unconditionally at boot.

Registration site	Count	Called from
Metrics::new, core daemon	120	main.rs:488
crux_mcp::ledger::register_metrics	10	main.rs:832
SessionMetrics::new	11	main.rs:833
redaction::register_metrics	1	main.rs:489

A naming inconsistency worth knowing before you write a dashboard query: build_info has no prefix, crux_witness_unwitnessed_heads uses crux_, the eleven session-plane metrics use vaultcrux_, and everything else uses corecrux_.

9.3 Core daemon metrics (1 to 60)

All registered in metrics.rs; the line is the constructor line.

#	Metric	Type	Labels	Measures	Registered
1	build_info	GaugeVec	version, commit, service	Build metadata	:161
2	corecrux_build_info	GaugeVec	version, commit, service, sdkVersion	Build metadata, hardening contract	:175
3	corecrux_io_backend	GaugeVec	backend	Selected IO backend	:195
4	corecrux_peer_cache_hits_total	Counter	-	Peer cache hits, sealed immutable blocks only	:204
5	corecrux_peer_cache_misses_total	Counter	-	Peer cache misses	:213
6	corecrux_peer_cache_bytes	Gauge	-	Peer cache size, best-effort	:222
7	corecrux_tail_cache_hits_total	CounterVec	shard	Tail cache hits	:231
8	corecrux_tail_cache_misses_total	CounterVec	shard	Tail cache misses	:240
9	corecrux_tail_cache_bytes	GaugeVec	shard	Tail cache resident bytes	:249
10	corecrux_valve_pause_ingest	Gauge	-	Operator valve state	:258
11	corecrux_http_inflight	Gauge	-	HTTP requests past the concurrency gate	:264
12	corecrux_http_rate_limited_total	CounterVec	key_kind	Requests rejected by the rate limiter	:273
13	corecrux_valve_pause_compaction	Gauge	-	Operator valve state	:285
14	corecrux_valve_throttle	Gauge	-	Operator valve state	:294
15	corecrux_valve_read_only	Gauge	-	Operator valve state	:300
16	corecrux_valve_emergency_brake	Gauge	-	Operator valve state	:306
17	corecrux_valve_state	GaugeVec	valve	One series per valve	:315
18	corecrux_throttle_ratio	Gauge	-	Token-bucket fullness, 1 means no pressure	:324
19	corecrux_data_dir_bytes_total	Gauge	-	Data directory total bytes	:333
20	corecrux_data_dir_bytes_free	Gauge	-	Data directory free bytes	:339
21	corecrux_data_dir_free_ratio	Gauge	-	Alert on this one. Free ratio	:345
22	corecrux_write_confirmations_total	CounterVec	signed	Write confirmations by signed state	:351
23	corecrux_write_confirmation_sign_duration_ms	Histogram	-	Write-confirmation signing latency	:363
24	corecrux_write_confirmation_unsigned_queue_depth	Gauge	-	Unsigned confirmations pending re-sign	:372
25	corecrux_tenant_throttle_rejected_total	CounterVec	tenant_id_hash	Tenant throttle rejections	:381
26	corecrux_emergency_brake_total	CounterVec	source	Emergency-brake activations	:393
27	corecrux_write_rejects_total	CounterVec	reason	Write rejects by reason	:405
28	corecrux_backpressure_active_gauge	Gauge	-	Backpressure active state	:417
29	corecrux_replay_total	CounterVec	result	Replay attempts	:424
30	corecrux_replay_mismatch_total	CounterVec	drift_class	Replay mismatches by drift class	:433
31	corecrux_segment_corrupt_total	CounterVec	reason	Detected segment corruption	:442
32	corecrux_verify_store_seconds	Histogram	-	verify-store run duration	:454
33	corecrux_segment_scrub_seconds	Histogram	-	Segment scrub run duration	:463
34	corecrux_dir_l0_runs	GaugeVec	shard	Directory L0 run count	:472
35	corecrux_dir_level_bytes	GaugeVec	shard, level	Directory run bytes per level	:481
36	corecrux_dir_compactions_total	CounterVec	shard, level_from, level_to, status	Directory compactions	:493
37	corecrux_dir_compaction_seconds	HistogramVec	shard, level_from, level_to	Compaction duration	:505
38	corecrux_dir_compaction_bytes_in_total	CounterVec	shard	Bytes read as compaction input	:517
39	corecrux_dir_compaction_bytes_out_total	CounterVec	shard	Bytes published as output	:529
40	corecrux_dir_dead_extent_ratio	GaugeVec	shard	Dead extent ratio	:541
41	corecrux_checkpoints_installed_total	CounterVec	shard, stream_type	Checkpoint installs	:553
42	corecrux_checkpoint_min_live_seq	GaugeVec	shard, stream_type	Latest installed min_live_seq	:565
43	corecrux_stream_tombstones_total	CounterVec	shard	Stream tombstones installed	:577
44	corecrux_stream_tombstone_rejects_total	CounterVec	shard	Appends rejected on a tombstoned stream	:589
45	corecrux_append_latency_seconds	HistogramVec	shard	Append latency	:601
46	corecrux_stream_read_latency_seconds	HistogramVec	shard, op	Stream read latency	:613
47	corecrux_read_retry_total	CounterVec	op, reason, outcome	Read retries. Feeds the read_retry_failed_threshold readiness gate	:625
48	corecrux_store_lock_wait_seconds	HistogramVec	op	Time waiting for the store lock	:634
49	corecrux_store_lock_hold_seconds	HistogramVec	op	Time holding the store lock	:646
50	corecrux_store_service_seconds	HistogramVec	op	Store service time excluding lock wait	:658
51	corecrux_append_lane_waiters	Gauge	-	Appends waiting for a lane lock	:670
52	corecrux_append_lane_waiters_peak	Gauge	-	Peak concurrent lane waiters	:679
53	corecrux_append_lane_queue_depth	Histogram	-	Lane queue depth at enqueue	:688
54	corecrux_append_lane_selected_total	CounterVec	bucket	Appends selected into fairness buckets	:697
55	corecrux_append_lane_wait_seconds_by_bucket	HistogramVec	bucket	Lane wait time by bucket	:709
56	corecrux_grpc_messages_sent_total	CounterVec	rpc	gRPC response messages sent	:721
57	corecrux_grpc_send_seconds	HistogramVec	rpc	gRPC send and encode duration	:733
58	corecrux_grpc_send_blocked_seconds	HistogramVec	rpc	gRPC send blocking duration	:742
59	corecrux_replay_events_total	CounterVec	rpc	Replay events returned	:754
60	corecrux_replay_bytes_total	CounterVec	rpc	Replay bytes returned	:763

The gRPC-labelled metrics exist but stay at zero in this build, because every gRPC RPC returns unimplemented, see chapter 1 §1.4.

9.4 Core daemon metrics (61 to 120)

#	Metric	Type	Labels	Measures	Registered
61	corecrux_replay_build_response_seconds	HistogramVec	rpc	Replay response materialisation time	:772
62	corecrux_replay_encode_seconds	HistogramVec	rpc	Replay protobuf encode sampling	:784
63	corecrux_rpc_total_seconds	HistogramVec	rpc	Total server-side time per RPC	:796
64	corecrux_storage_tail_stage_seconds	HistogramVec	stage	Tail-read stage duration: index_lookup, io, decode, total	:808
65	corecrux_storage_append_stage_seconds	HistogramVec	stage	Append stage duration: idempotency_check, index_update, io_write, fence_wait, fence_fsync, fence, total	:820
66	corecrux_append_fence_wait_seconds	HistogramVec	shard	Append durability-fence wait	:832
67	corecrux_append_fence_fsync_seconds	HistogramVec	shard	Append fsync time	:844
68	corecrux_storage_tail_bytes_total	CounterVec	kind	Tail-read bytes: disk_estimate, frame	:856
69	corecrux_storage_tail_items_total	CounterVec	kind	Tail-read items touched: segments, blocks, frames	:868
70	corecrux_storage_tail_path_total	CounterVec	path, outcome	Tail-read fast-path outcomes	:880
71	corecrux_storage_head_frames_scanned_total	Counter	-	Head frames inspected while serving tail reads	:892
72	corecrux_read_amplification_p50	GaugeVec	shard	Read amplification p50, rolling	:901
73	corecrux_read_amplification_p95	GaugeVec	shard	Read amplification p95, rolling	:913
74	corecrux_kernel_launch_total	CounterVec	kernel, result	Kernel launches and outcomes	:925
75	corecrux_shardmap_version	Gauge	-	Shard-map version loaded by this process	:934
76	corecrux_routing_lookup_total	CounterVec	op, outcome	Routing lookups	:943
77	corecrux_routing_lookup_seconds	HistogramVec	op	Routing lookup duration	:952
78	corecrux_shard_requests_total	CounterVec	shardId, op	Requests routed to a shard	:961
79	corecrux_replication_receive_total	CounterVec	result	Replication segment receive and apply outcomes	:973
80	corecrux_replication_follower_watermark_segment_seq	GaugeVec	shardId	Follower-applied highest segment_seq	:985
81	corecrux_replicated_commit_total	CounterVec	result	ReplicatedCommit outcomes	:997
82	corecrux_replicated_commit_required_acks	GaugeVec	shardId	Required acknowledgements	:1006
83	corecrux_replicated_commit_actual_acks	GaugeVec	shardId	Observed acknowledgements	:1018
84	corecrux_replicated_commit_ack_deficit	GaugeVec	shardId	Required minus actual	:1030
85	corecrux_replication_shard_epoch	GaugeVec	shardId	Current shard epoch	:1042
86	corecrux_replication_follower_targets	GaugeVec	shardId	Configured follower count, excluding self	:1054
87	corecrux_replication_topology_ok	GaugeVec	shardId	Topology sanity	:1066
88	corecrux_replication_leader_segment_seq	GaugeVec	shardId	Latest leader segment_seq for shipping	:1078
89	corecrux_replication_min_follower_acked_segment_seq	GaugeVec	shardId	Minimum follower-acked segment_seq	:1090
90	corecrux_replication_lag_segments	GaugeVec	shardId	Lag in segment_seq units	:1102
91	corecrux_shard_state	GaugeVec	shardId, state	Shard state one-hot: active, draining, retired	:1114
92	corecrux_projections_commit_id	GaugeVec	shard	Latest projections commit_id	:1126
93	corecrux_projections_cursor_segment_seq	GaugeVec	shard, projection	Projection cursor segment_seq	:1138
94	corecrux_projections_cursor_offset	GaugeVec	shard, projection	Projection cursor offset	:1150
95	corecrux_projections_row_count	GaugeVec	shard, projection	Committed projection row count	:1162
96	corecrux_projections_tick_frames_total	CounterVec	shard	Frames processed by projection ticks	:1174
97	corecrux_projections_tick_seconds	HistogramVec	shard	Projection tick duration	:1186
98	corecrux_projections_tick_fail_total	CounterVec	shard	Projection tick failures	:1198
99	corecrux_shard_open_attempts_total	CounterVec	caller	ShardStorage::open() calls by caller context	:1210
100	corecrux_lock_contention_total	CounterVec	caller	File-lock contention events	:1222
101	corecrux_projection_snapshot_valid	GaugeVec	projection	Snapshot validity per required projection. Set on every /readyz call	:1234
102	corecrux_knowledge_authority_mode	GaugeVec	mode	Knowledge authority mode one-hot	:1246
103	corecrux_knowledge_rollout_stage	GaugeVec	stage	Knowledge rollout stage one-hot	:1258
104	corecrux_knowledge_parity_status	GaugeVec	status	Last knowledge-parity status one-hot	:1270
105	corecrux_knowledge_rollback_triggered	Gauge	-	Rollback trigger active	:1282
106	corecrux_knowledge_parity_mismatch_count	Gauge	-	Last observed parity mismatch count	:1291
107	corecrux_knowledge_parity_cursor_missing_count	Gauge	-	Last observed missing-cursor count	:1300
108	corecrux_knowledge_parity_pass_ratio_bps	Gauge	-	Last observed parity pass ratio, basis points	:1309
109	corecrux_knowledge_parity_projection_lag_ms	Gauge	-	Last observed parity projection lag	:1318
110	corecrux_receipt_verify_total	CounterVec	result	Receipt signature verification outcomes	:1327
111	corecrux_receipt_verify_fail_total	CounterVec	reason	Receipt verification failures by reason	:1339
112	corecrux_receipt_export_total	CounterVec	status	Receipt export bundle requests	:1351
113	corecrux_query_graph_expand_duration_seconds	Histogram	-	Graph-expand query duration	:1364
114	corecrux_query_graph_expand_nodes_visited	Histogram	-	Nodes visited per graph-expand query	:1376
115	corecrux_query_time_range_duration_seconds	Histogram	-	Time-range query duration	:1388
116	corecrux_query_time_range_artifacts_scanned	Histogram	-	Artifacts scanned per time-range query	:1400
117	corecrux_seal_duration_seconds	HistogramVec	phase	Time to seal a segment, including the .ccxi build	:1413
118	corecrux_seal_backlog_frames	Gauge	-	Frames in the head segment not yet sealed	:1426
119	corecrux_ccxi_missing_total	Gauge	-	Sealed segments missing a .ccxi companion	:1435
120	crux_witness_unwitnessed_heads	Gauge	-	Seal-chain heads sealed but not yet witnessed	:1444

MCP tool-ledger metrics (121 to 130)

Registered in ledger.rs.

#	Metric	Type	Labels	Measures	Registered
121	corecrux_tool_invocation_duration_seconds	HistogramVec	tool, outcome	MCP tools/call dispatch latency	:246
122	corecrux_token_spend_total	CounterVec	tool	Estimated tokens per tool, arguments plus result	:257
123	corecrux_tool_response_truncated_total	CounterVec	tool, reason	Responses truncated by a budget-honouring path	:265
124	corecrux_tool_ledger_emit_failures_total	CounterVec	reason	Ledger observation appends that failed. Never fails the tool call	:273
125	corecrux_coverage_events_without_receipt	IntGauge	-	Events in the last attested window with no receipt	:281
126	corecrux_coverage_receipts_without_anchor	IntGauge	-	Receipt bodies with no external anchor	:286
127	corecrux_coverage_gaps_total	IntGauge	-	Total gaps	:291
128	corecrux_coverage_events_total	IntGauge	-	Events covered by the last attested window	:296
129	corecrux_coverage_receipts_total	IntGauge	-	Receipts covered by the last attested window	:301
130	corecrux_coverage_anchored_total	IntGauge	-	Anchored receipts in the last attested window	:306

Session-plane metrics (131 to 141)

Registered in session_metrics.rs. These use the vaultcrux_ prefix.

#	Metric	Type	Labels	Measures	Registered
131	vaultcrux_session_handshakes_total	CounterVec	origin, outcome	Session handshake requests	:34
132	vaultcrux_session_handshake_latency_seconds	HistogramVec	origin	End-to-end handshake latency	:46
133	vaultcrux_session_capability_graph_size	HistogramVec	origin, tier	Capabilities in issued session plans	:59
134	vaultcrux_session_active	Gauge	-	Currently-active sessions in the local registry	:72
135	vaultcrux_session_expired_total	CounterVec	origin, reason	Sessions removed: ttl_expired, client_closed, admin_closed	:79
136	vaultcrux_session_plan_bytes	HistogramVec	encoding	Size of issued session plans	:91
137	vaultcrux_invocation_receipts_total	CounterVec	channel, capability, outcome	Per-capability invocation receipt counts	:101
138	vaultcrux_invocation_receipt_latency_seconds	HistogramVec	channel, capability	Per-capability invocation latency	:113
139	vaultcrux_invocation_verify_total	CounterVec	outcome	POST /invocation/verify outcomes	:126
140	vaultcrux_session_plan_sealer_errors_total	Gauge	-	Cumulative segment-seal errors during session mint	:135
141	vaultcrux_session_segment_seal_failures_total	Gauge	-	Cumulative always-store seal failures that failed a handshake closed	:144

Log-redaction metric (142)

#	Metric	Type	Labels	Measures	Registered
142	corecrux_log_redactions_total	CounterVec	rule	Redaction-rule hits at the log sink. Increments in both on and audit modes, which makes it the pre-flight signal before switching to on	:53

Registration is idempotent, a double registration is logged at debug and ignored.

9.5 Tracing and OpenTelemetry

Property	Value
Cargo feature	otel, not in default (Cargo.toml:10-16)
Endpoint	OTEL_EXPORTER_OTLP_ENDPOINT (main.rs:2085)
Transport	OTLP over gRPC (tonic) (main.rs:2087)
Exporter	Batch span exporter (main.rs:2093)
Resource	service.name = "corecruxd", hard-coded, not CORECRUXD_SERVICE (main.rs:2094)
Propagator	W3C TraceContextPropagator, set globally (main.rs:2103)
Shutdown	Flushed on SIGINT and SIGTERM before the shutdown broadcast (main.rs:2147)

Behaviour matrix:

Build	Endpoint set?	Result
default, no otel	anything	The whole block is compiled out (main.rs:2077). Plain fmt subscriber; the variable is ignored
--features otel	unset	Falls through to the same plain subscriber. No OTLP layer, no error
--features otel	set, exporter builds	Logs and spans
--features otel	set, exporter build fails	Silently falls through to the plain subscriber (main.rs:2087). A typo'd endpoint produces no diagnostic

In the otel path the log format is still selected by LOG_FORMAT, and the redacting writer still applies.

You can correlate traces without compiling otel. Even in the default build, x-request-id and traceparent are extracted, sanitised and surfaced in the structured operations log (structured_log.rs:85). Distributed-trace correlation by log join is available out of the box.

9.5.1 corruption_state_clear is a one-way latch, and only a restart clears it

Gate 7 deserves its own note, because the obvious assumption about it is wrong and the consequence is a permanent outage.

corruption_detected is in-process memory, not a file and not a database row, an Arc<RwLock<bool>> on the shared application state (mod.rs:366), initialised to false at boot (main.rs:598).

Exactly two code paths write it, both in the admin plane and both setting it to true (admin.rs:505, admin.rs:557). No production code path anywhere sets it back to false.

And in the Community Edition it cannot be tripped at all. Both set-sites sit behind a dataplane pool that this edition hard-wires to None (main.rs:565), so verify-store and scrub return dataplane disabled (admin.rs:508) and never reach the line that sets the flag. Gate 7 therefore reports clear on a CE daemon because nothing can set it, not because the store has been checked. Do not read a green gate 7 here as evidence of integrity; run corecruxctl verify-store --strict and read its output.

Where the dataplane IS wired, the honest operational statement is:

• Once verify-store or scrub sets the flag, /readyz returns 503 for the lifetime of the process.
• There is no endpoint, CLI command or configuration change that clears it. Any documentation describing this gate as "operator-cleared", including earlier drafts of this table, is wrong.
• The only way out is to restart the daemon, which resets the flag to false because it is process-local.

That last point cuts both ways, and an operator needs both halves. A restart clears the alarm without repairing the data: the flag says a scrub found corruption, and restarting throws that finding away. Run corecruxctl verify-store --strict and resolve what it reports before restarting, or you will have silenced the only signal you had.

9.6 Health, readiness and the nine gates

The daemon exposes exactly three probe endpoints: /healthz, /readyz and /metrics (http/mod.rs:473-475). There is no /livez and no /startupz. All three are unauthenticated. /v1/version is also public; /v1/admin/version requires admin:read.

CORECRUXD_PUBLIC_PROBES_MINIMAL (default off) makes /healthz omit routing and valves, and makes a /readyz failure return {"ok":false,"checks":[]} with the per-gate breakdown withheld (health.rs:64). It does not cover /metrics.

GET /healthz: liveness

Always 200, always ok: true. Nothing in the handler can make it fail (health.rs:23). It is a pure liveness signal, do not use it as a readiness probe.

{
  "ok": true,
  "build":   { "version": "…", "commit": "…" },
  "compat":  { "requires": "…" },
  "sdkVersion": "…",
  "routing": { "shardMapVersion": 1, "shardCount": 4, "lastReloadAt": "…", "nodeId": "…" },
  "valves": {
    "pause_ingest":     { "enabled": false, "actor": "", "reason": "", "updatedAtUnixNs": 0 },
    "pause_compaction": { "…": "…" },
    "throttle":         { "…": "…" },
    "read_only":        { "…": "…" },
    "emergency_brake":  { "…": "…" }
  }
}

In minimal mode routing and valves are omitted.

GET /readyz: readiness

Success is 200 with {"ok": true} (health.rs:270). Failure is 503:

{
  "ok": false,
  "checks": [
    { "name": "data_dir_capacity", "ok": false,
      "error": "data dir free ratio below emergency threshold (free_ratio=0.043 threshold=0.100 free_bytes=37580963840 total_bytes=879609302220)" }
  ]
}

Only failing gates appear in checks, a passing gate is never listed. In minimal mode the array is emptied but the 503 and ok:false remain.

Every readiness gate

Evaluated in this order; all nine must pass for a 200 (health.rs:260).

#	name	Passes when	Threshold	Failure error	Source
1	data_dir_lock_held	The <data_dir>/LOCK flock is held	structural	LOCK file not held	health.rs:274
2	routing_loaded	The shard map is non-empty	none	routing table not loaded	health.rs:281
3	replicated_commit_dataplane	Not in ReplicatedCommit, or a dataplane pool exists. Always fails in this build if CORECRUXD_COMMIT_LEVEL=replicated_commit, because the pool is hard-wired to None (main.rs:565)	CORECRUXD_COMMIT_LEVEL, default local_commit	replicated commit selected but dataplane store is unavailable	health.rs:288
4	replicated_commit_topology	Under ReplicatedCommit only: every non-retired shard this node leads has at least one other follower	shard map contents	replicated commit requires followers; <N> local leader shard(s) missing followers: …	health.rs:295
5	read_retry_failed_threshold	Failed context-lost read retries are below the threshold, or the threshold is 0	CORECRUXD_READ_RETRY_FAILED_READYZ_THRESHOLD; 0 disables	failed read retries exceeded threshold (failed=<n> threshold=<t>)	health.rs:302
6	projection_snapshots_valid	No dataplane pool, always true in this build, or no snapshot issues. Side effect: sets corecrux_projection_snapshot_valid for four projections on every call	none	projection snapshots invalid (…), first four issues then a count	health.rs:309
7	corruption_state_clear	No corruption flag set by verify-store or scrub	restart only, see below	corruption state set by verify-store/scrub	health.rs:316
8	control_evidence_ok	Control evidence is not hosted locally, or its verification passed at boot	reconciled at main.rs:567	the recorded error, else control evidence verification failed	health.rs:323
9	data_dir_capacity	Measurement succeeded, total is above zero, and free_ratio >= emergency_free_ratio	CORECRUXD_CAPACITY_EMERGENCY_FREE_RATIO, default 0.10, 10% free	the measurement error, else data dir free ratio below emergency threshold (…)	health.rs:330

Gate 9 in full: the one that bites

A data partition below 10% free takes an otherwise-healthy daemon out of rotation. That is correct behaviour and it is also the single most confusing failure in practice, because everything downstream fails with an unhelpful message. Integration tests against such a daemon fail with a bare "not healthy in 10s" and empty stderr; orchestrators mark the pod unready with no application-level error; unrelated features appear broken.

When something inexplicable fails, check df -h on the data partition first.

Measurement is fs2::total_space and fs2::available_space on config.data_dir (main.rs:2436), available space for this user, not raw free space. It is refreshed by the background capacity guard.

Env var	Meaning	Default	Clamp
CORECRUXD_CAPACITY_GUARD_ENABLED	Run the background guard	on	-
CORECRUXD_CAPACITY_GUARD_INTERVAL_SECS	Re-measure interval	30	10..=3600
CORECRUXD_CAPACITY_WARNING_FREE_RATIO	Warning level	0.20	0.01..=0.95
CORECRUXD_CAPACITY_CRITICAL_FREE_RATIO	Critical level	0.10	0.01..=0.90
CORECRUXD_CAPACITY_EMERGENCY_FREE_RATIO	The /readyz gate threshold	0.10	0.01..=0.90
CORECRUXD_CAPACITY_RESUME_FREE_RATIO	Auto-resume after an auto-pause	0.20	0.02..=0.99

The four ratios are re-ordered after parsing so they cannot be inconsistent (config.rs:1117). Raising EMERGENCY to 0.5 while leaving WARNING at the default raises warning to 0.5 as well.

Three gauges are updated on every guard tick: corecrux_data_dir_bytes_total, corecrux_data_dir_bytes_free and corecrux_data_dir_free_ratio. Alert on corecrux_data_dir_free_ratio above the emergency threshold and you get warning before /readyz flips.

If the measurement itself fails, the path is gone, for example, the guard sets total, free and ratio to zero and records the error, so gate 9 fails with the measurement error rather than a ratio message.

Note also that the emergency threshold drives the capacity guard's autonomous write to CONTROL.json. See chapter 6 §6.8.

9.7 Version endpoints

GET /v1/version is public (health.rs:491). Body keys: version, msrv, product, cloud_access, agent_workbench, features (text_search, graph_expand, self_observe, mcp, embeddings), capabilities (coordination, consolidation_scheduler, context_surface, local_ingest, auto_capture, status_feed, activity_log, each {enabled: bool}), semantic_profile, protocol_contracts, sync, update.

Three things are deliberately withheld from the public payload (health.rs:516): the build commit, the sync remote_url (the public body exposes only remote_url_redacted: bool), and update commit SHAs and repository directories.

GET /v1/admin/version requires admin:read (health.rs:597) and is a superset: it adds commit, passport{fingerprint, public_key_hex, alg}, cloud, action_enrichment, gpu1_compute, the full sync.remote_url and the full update view.

passport.public_key_hex there is the verification key for every receipt this daemon mints. An auditor holding the observations/*.jsonl files plus that hex can verify offline (health.rs:633).

cloud_access.contract_path and gpu1_compute are null in a stock build, both are hosted-surfaces-gated.

9.8 What observability does not give you

• No JSON logs from any shipped manifest, because they all set the wrong variable name. Set LOG_FORMAT=json yourself.
• No redaction by default, audit counts and ships.
• No authentication on /metrics, and it leaks shard ids, node topology, valve state and hashed tenant ids.
• No diagnostic when the OTLP exporter fails to build.
• No /livez and no /startupz. /healthz cannot fail, so it is useless as a readiness signal; use /readyz.
• No log line at all for the first nine boot steps, including every config-parse and auth-posture decision.
• No metric for the fact journal's size, watch the disk, not a counter.

Sources

• crates/corecruxd/src/main.rs:2065, init_tracing
• crates/corecruxd/src/main.rs:2068, the LOG_FORMAT read
• crates/crux-observe/src/redact.rs:68, CORECRUXD_REDACT parsing
• crates/corecruxd/src/metrics.rs:159, the Prometheus registry
• crates/corecruxd/src/http/health.rs:144, the /readyz handler
• crates/corecruxd/src/http/health.rs:260, the nine-gate evaluation
• crates/corecruxd/src/main.rs:2436, the capacity measurement
• crates/corecruxd/src/structured_log.rs:127, StructuredOpLog