CueCruxVaultCrux

Crux Daemon · 11. Retrieval and token budgets

query_facts is a case-insensitive substring OR-match over a fact's value, key and entity. BM25 exists, but only on the separate segment lane reached by query / query_scan / query_expand. fused_retrieve and graph expansion are library code whose only callers are tests, with the sparse weight hardwired to 0.0. And the default embedder is feature hashing of unigrams and bigrams into 256 dimensions, lexical overlap, not meaning.

Those four sentences are the whole answer, and they are stated first because everything else in this chapter is easier to read once you have them. What is there works, is deterministic, is fast, and is documented below in full.

This chapter is reference. The store beneath it is chapter 10; the flags named here are listed in full in chapter 5; defects rather than narrowness are in chapter 16.

11.0 In plain English

Retrieval is the half of memory that decides what comes back. Chapter 10 covers writing things down; this chapter covers the moment an agent asks a question and something has to pick which of the stored facts to hand over, in what order, and how many.

For the fact lane, which is the one nearly everybody is on, the honest mental model is grep with a ranking step, not a search engine. The daemon lowercases your query, looks for it as a substring inside each fact's value, key and entity, keeps the ones that contain it, and sorts what survives. There is a second, quite different lane over the binary segment files, and that one is a real inverted index with BM25 scoring. The two are reached by different tools and share almost nothing, which is why §11.1 spends a table separating them before anything else.

The other half of the chapter is token_budget, and the reason it exists is worth stating plainly. Every fact returned to an agent occupies part of a finite context window, and an unbounded recall over a store that has been running for months will happily return more than the model can hold. A budget is a spending limit on the answer, not a filter on the question: the daemon ranks everything that matched and then emits down the list until the budget is spent. Passing one on every retrieval call is the single most effective defence against a session that fills its context with recall and has no room left to work.

You will come here in two situations. Something you know is in the store did not come back, and you need to understand why; or your context is filling up and you want to know which lever actually controls it. For the first, start at §11.2 and check whether your query words literally appear in the stored text. For the second, read §11.8 and §11.9 together, and note in particular that get_bootstrap has no token_budget parameter at all: passing one is accepted, silently ignored, and up to 100 facts come back concatenated regardless.

The thing people get wrong is expecting meaning-based matching. Asking for "how do we deploy" will not find a fact whose value reads "release procedure", because no word in the query appears in the stored text. The default embedder does not rescue this either: it is feature hashing of unigrams and bigrams into 256 dimensions, which measures lexical overlap rather than meaning, and §11.5 explains why two different things in this system wear the word "dense". Write your facts with the words you will later search for, and this lane works well; expect it to infer synonyms, and it will disappoint.

11.1 There are two retrieval lanes and they share almost nothing

Fact laneSegment lane
Toolsquery_facts, get_bootstrap, memory_viewquery, query_scan, query_expand
Backing storefacts.jsonl replayed into in-memory maps.ccxseg segments plus .ccxi companions
Matchingcase-insensitive substring OR-matchBM25 over a real inverted index
Rankingdecay-adjusted confidence desc, then stored_at descBM25 score, optional dense re-rank
Dense vectorswired at the store level but unused by query_factsCosineDenseProvider over .ccxv companions
Coverage reportnonecoverage.{score, missing_tokens, below_floor}
StatusSHIPPEDSHIPPED

An agent following the common operator playbook, query_facts with a token_budget, is on the substring lane, not the BM25 lane. That is the single most consequential thing to know about retrieval in this daemon.

11.2 The fact lane, end to end

handle_query_facts (facts.rs:422) delegates to query_visible_facts_opts_as_of (facts.rs:719). Five stages, in this order.

  1. Filter (facts.rs:727): !deleted, then valid_at(as_of), then superseded_by.is_none() unless include_superseded, then scope::fact_visible_to_identity, then tenant match, then entity-prefix match, then entity match, then query match.
  2. Query match, fact_matches_query (facts.rs:807): lowercase the query, split on whitespace, and return true if any term is a substring of the lowercased value, key or visible entity.
  3. Rank (facts.rs:757): effective confidence descending, stored_at descending as tiebreak.
  4. Confidence floor (facts.rs:775), counting what it drops.
  5. Budget or top_k (facts.rs:787): greedy fill by fact.tokens until the budget would be exceeded, always admitting at least one; otherwise truncate to top_k, default 10 (facts.rs:425).

Stage 2 is the one that surprises people, so state it precisely: no stemming, no tokenisation beyond whitespace, no IDF, no phrase handling, and OR semantics. A two-word query matches a fact containing either word. Searching for deploy status returns every fact mentioning deploy and every fact mentioning status, unranked by how many terms hit.

There is no BM25, no embedding and no graph expansion on this path.

What it is good for: exact-ish recall over an entity/key namespace you control. If your agents write facts under disciplined entity prefixes (execplan:<slug>, bench:<id>, incident:<date>), the entity-prefix filter plus substring match is a fast, deterministic, zero-dependency lookup that never surprises you with a ranking model. Design your entity naming and it works well. Expect it to find semantically-related material and it will not.

11.3 The store's own query, and get_bootstrap

FactStore::query / query_inner (fact_store.rs:1301, :1362) is a different path from query_facts, and it does use embeddings when an embedder is present: it skips keyword filtering entirely (:1390) and sorts by 0.6 * cosine + 0.4 * confidence (:1414), falling back to confidence-then-recency when there is none (:1435).

This path serves internal and metadata lookups, passports (passport.rs:169), audit (audit.rs:122), coordination intents (coord.rs:130), and it serves get_bootstrap (facts.rs:849).

So the two most-used recall tools take different code paths with different semantics:

  • query_facts → substring OR-match, decay-ranked, budget-aware.
  • get_bootstrap → cosine-blended when an embedder is present (which it is by default, via LocalHashEmbedder), keyword-substring when it is not.

get_bootstrap has no token_budget parameter

Its input schema declares exactly two properties, topic and query (mod.rs:378). The handler hard-codes token_budget: None with top_k: 100 (facts.rs:838) and joins every returned fact into one text blob (facts.rs:877).

Operator guidance that says to call get_bootstrap(topic="patterns", token_budget=500) is passing an argument the tool does not accept and does not apply. The call succeeds; the budget is silently ignored; up to 100 facts come back concatenated. If bootstrap output is blowing your context, the lever is the topic filter and the size of your __bootstrap__:: namespace, not a budget argument.

memory_view is a third path again: it iterates all_facts() directly with a reserved-prefix filter (memory.rs:46), sorts newest-first with confidence as tiebreaker (memory.rs:209), and does honour token_budget (memory.rs:154).

11.4 The segment lane: BM25 and coverage

bm25_search (bm25.rs:233) is Okapi BM25 with k1 = 1.2 and b = 0.75 (bm25.rs:19) and Robertson–Sparck-Jones IDF:

idf     = ln((N - df + 0.5) / (df + 0.5) + 1.0)                      bm25.rs:293
tf_norm = (tf * (k1 + 1)) / (tf + k1 * (1 - b + b * dl / avg_dl))    bm25.rs:306
score  += idf * tf_norm

df is computed globally across all readers (bm25.rs:285), so multi-segment scoring is IDF-correct, at the cost of re-decoding postings once per reader per term. Tenant filtering checks the full 64-bit hash with lo16 as a fast precheck (bm25.rs:25).

This is real information retrieval and it is worth using. It is also the lane most operators never reach, because it requires sealed segments from the local ingest path.

The coverage report

Computed at bm25.rs:341.

FieldMeaning
coverage.scorequery tokens that matched at least one document in any reader, divided by total query tokens, a float in [0,1]
coverage.missing_tokensthe unmatched tokens
coverage.below_floorthe count of hits dropped by an explicit min_score (bm25.rs:331); 0 when no floor is set

The coverage field is named differently on the two surfaces, and shaped differently too. This catches people, because the daemon's own bootstrap playbook documents one form and the MCP tool returns the other. Both are correct; neither is a typo.

SurfaceFieldShape
HTTP POST /v1/query/text-searchcoverage.gaps[]array of objects: {query_terms, match_quality, suggestion} (query.rs:634)
MCP query toolcoverage.missing_tokensflat array of strings (query.rs:145)

The underlying struct field is missing_tokens. The HTTP route transforms it on the way out, one object per unmatched term, with a human-readable suggestion; the MCP tool passes it through raw. So a client written against coverage.gaps[] will find nothing under that name over MCP, and a client written against missing_tokens will find nothing under that name over HTTP, and in both cases the failure is a silently absent field rather than an error.

Do not confuse either with get_gaps in the MCP surface, which is a Feature Registry capability-gap tool (mod.rs:1025) and has nothing to do with retrieval coverage.

One caveat worth knowing before you build an alert on it: missing_tokens is reconstructed positionally, by mapping token_matched[i] back onto the original query string split on non-alphanumerics with words longer than one character (bm25.rs:350). If the analyzer's tokenisation differs from that naive split, the names in missing_tokens can misalign with the tokens that actually missed. The score is reliable; the token names are best-effort.

11.5 Dense: two different things wear the word

(a) The fact-store embedder

main.rs selects one of three in priority order (main.rs:900):

  1. CORECRUXD_EMBED_DELEGATE_URL set, a DelegatingEmbedder, authenticated daemon-to-daemon, mutually exclusive with CORECRUXD_EMBEDDING_URL (config.rs:625).
  2. CORECRUXD_EMBEDDING_URL set, an external HTTP embedder with dimensions: 0 for auto-detect (main.rs:930).
  3. Neither set, and CORECRUXD_LOCAL_EMBEDDER not explicitly disabled, LocalHashEmbedder (main.rs:942). local_embedder_enabled defaults to true when unset (config.rs:1280).

The default "dense" lane is not semantic. LocalHashEmbedder (embeddings.rs:830) is feature hashing: lowercase the text, split on non-alphanumerics, hash each unigram and each adjacent bigram with BLAKE3, take the first four bytes mod 256 as the bucket index, add ±1 by a sign bit from byte four, then L2-normalise. Model id crux-local-hash-v1, 256 dimensions (embeddings.rs:841).

Its own doc comment states the property exactly: lexically overlapping texts score high cosine. It has no notion of synonymy and no notion of meaning. Two documents that say the same thing in different words score near zero. That is not a defect; it is a deliberate zero-dependency, zero-download, fully-offline default, and for near-duplicate detection and typo-tolerant matching it does the job. It is simply not what "semantic search" means.

The upgrade path is CORECRUXD_DENSE_MODEL=fastembed with the daemon built --features dense-embed-model, which uses all-MiniLM-L6-v2 at 384 dimensions (embeddings.rs:915). The default build does not compile or download it, the source says so at embeddings.rs:919, and if initialisation fails the daemon silently falls back to LocalHashEmbedder (main.rs:960). Check the model id in your fact embeddings if you need to know which one you are running.

One further wrinkle: when the embedder is a DelegatingEmbedder, fact enrichment is skipped entirely (fact_store.rs:1087). Delegated embedding is a prose and index lane only, so fact recall stays lexical (fact_store.rs:1315).

(b) The segment dense re-rank

handle_query (query.rs:63) applies a dense re-rank only when all of the following hold: a dense-provider factory is wired into the MCP context, the fact store has an embedder, the query embeds successfully, and the corpus has .ccxv vector companions. The fusion is a fixed 0.7 * bm25_normalised + 0.3 * cosine (query.rs:60). Any missing piece degrades to bit-identical BM25 (query.rs:56).

The result declares its own score_space, bm25_dense_fused or bm25_lexical, and a score_merge_rule (query.rs:85). Read that field. It is the daemon telling you, per response, which algorithm actually ran, and it is the right way to answer "am I getting dense re-ranking?" without guessing from configuration.

11.6 Fusion and graph expansion are declared, not wired

corecrux-retrieval/src/fused.rs implements fused_retrieve with FusionWeights { bm25: 0.5, graph: 0.3, dense: 0.2, sparse: 0.0 } (fused.rs:41), a graph cold-start override that zeroes the graph weight and redistributes it to BM25 when the projection graph has fewer than 100 entities (fused.rs:81), and a DenseProvider hook.

It has no production call site. Every reference inside corecruxd is in a #[cfg(test)] module, local_ingest.rs:972 and codegraph_fusion.rs:142 are both use statements inside test modules. The production MCP surface calls bm25::bm25_search directly (query.rs:45, reuse.rs:76) and so does the HTTP surface (http/query.rs:496). corecrux-retrieval/src/graph.rs likewise has no non-test caller in either corecruxd or crux-mcp.

And within the formula itself, the sparse term is multiplied by a literal 0.0 (fused.rs:235) with the comment // future: learned sparse.

Status: DECLARED-NOT-WIRED. Three-lane fusion, including graph expansion, is library code with tests and no production path. CORECRUXD_QUERY_GRAPH_EXPAND is a separate dataplane-query flag documented in config.example.env as opt-in; it does not activate fused.rs.

11.7 What is actually in the default path

With CORECRUXD_AUTH_MODE=off and nothing else set, the config.example.env starting posture:

SurfaceWhat runs
query_factssubstring OR-match, ranked by decay-adjusted confidence then recency, budget-trimmed. No BM25, no vectors, no graph
queryBM25 (k1=1.2, b=0.75) over .ccxi companions, but only if CORECRUXD_BUILD_CCXI=1 produced them (config.example.env ships =1) and the local ingest lane has sealed segments. Dense re-rank only fires with a wired provider factory and .ccxv companions
get_bootstrapcosine-blended over LocalHashEmbedder vectors; that is, lexical-overlap cosine
Salienceinert (CORECRUXD_MEMORY_SALIENCE off)
Graphabsent

11.8 How a token is counted

Two estimators, and they disagree with each other and with your model provider.

The module is explicit that precision is a non-goal and comparability is the point (token_estimate.rs:13); there is no tokenizer dependency, so these are not BPE token counts.

Say it plainly: token_budget is enforced against a roughly-4-characters-per-token heuristic, not a model tokenizer. Budget numbers are comparable to each other across calls to the same daemon. They are not comparable to a provider's usage report, and a budget of 2000 will not land on exactly 2000 tokens in any model's counting.

11.9 Enforcement: three shapes

(a) Fact path, legacy drop. Greedy fill by fact.tokens, stopping before the first fact that would exceed the budget, always admitting at least one; overflow facts are dropped (facts.rs:787, mirrored at fact_store.rs:1446).

(b) Segment path, reversible pointer budget. The segment query response is pointer-only, metadata, no document text, so the budget is charged at the pointer price rather than the full-document hydration price (query.rs:100). POINTER_TOKENS = 40 (budget.rs:37) and pointers_within_budget(budget) = max(budget / 40, 1) (budget.rs:41). A budget of 500 admits 12 pointers where the legacy full-document take_while admitted about 2, the module header documents this as a 6× recall lift at the same budget (budget.rs:25).

(c) Fact path, tiered reversible. Active when token_budget is set, the CRC-v1 contract is negotiated, and the request is not a holdout control (facts.rs:480). The store-level budget drop is suppressed so the full ranked set arrives, then fact_emit_within_budget (budget.rs:79) computes:

full_count = fact_full_within_budget(costs, budget)    // greedy, >= 1
used_full  = sum(costs[0..full_count])
epitomes   = (budget - used_full) / POINTER_TOKENS
emit_count = min(full_count + epitomes, len)

The leading full_count facts are hydrated in full; the next emit_count - full_count are demoted to epitome-only pointers; everything past emit_count is dropped and the true total is disclosed as total_candidates (facts.rs:582). Nothing is silently lost, the count of what did not fit is always in the response.

11.10 Pointers, and what query_expand returns

A demoted result becomes an epitome pointer in the CRC-v1 envelope. Re-expansion differs by lane.

Segment lane. The handle is result_id = "{segment_index}:{doc_id}" (query.rs:126) and re-expansion is query_expand with result_ids (query.rs:255). There is no TTL and no server-side cursor state, the handle is a positional address into the loaded reader set. Three typed failures, which a client should distinguish:

ErrorMeaningWhat to do
evicted (query.rs:296)the segment is gonere-query; do not retry the expand
tenant_mismatch (query.rs:314)the handle belongs to another tenantdo not retry
invalid_format (query.rs:280)malformed handlefix the client

Fact lane. The pointer carries a content_hash, FNV-1a plus a splitmix64 finaliser rendered as 16 hex characters (budget.rs:94). It is change-detection only and explicitly not security-sensitive (budget.rs:92). The agent re-addresses the fact by (entity, key); a hash mismatch means the value changed, and "no facts found" means it was forgotten.

query_expand returns metadata, not text

In this repository, expansion returns segment_index, doc_id, frame_offset, token_count and a tokens_loaded total (query.rs:326). It does not return document text. The reason is stated in the source: "text lives in the dataplane frame store, out of this repo" (budget.rs:15).

So the two-pass path in this build is scan → address, not scan → read. If you were planning to use query_expand to pull document bodies into a prompt, it will not do that here.

11.11 The scan → expand two-pass path

query_scan (query.rs:171) runs the same bm25_search with min_score: None, then walks hits accumulating doc_length_tokens, stopping before the budget would be exceeded (always returning at least one) and setting budget_truncated: true (query.rs:200). It returns {scan, total_candidates, tokens_returned, budget_truncated, meta}.

Note the asymmetry: query_scan charges the full-document token cost, not the pointer price. It is the honest bounded-cost variant. query is the recall-optimised one. Choose query_scan when you need a hard ceiling on what a survey will cost you, and query when you want maximum candidates for a fixed budget.

11.12 Token savings, and the number that argues against us

token_savings (token_savings.rs:32) reports two numbers from two estimators, and the code insists they not be conflated (token_savings.rs:51).

The compaction arm is paired and structurally non-negative. sample_compaction (holdout.rs:289) serialises the same value twice, pretty and compact, and records the paired chars-over-four counts. paired_savings (holdout.rs:109) takes per-pair (control − treatment)/control, skips zero-control pairs, and reports a mean with a 1.96·SE interval. Compact JSON is never longer than pretty JSON of the same value, so this arm cannot go negative.

The net arm is unpaired live traffic, and it can be negative. Arms are assigned deterministically by FNV-1a plus splitmix64 bucketing (holdout.rs:49), no RNG, so a given request key always lands in the same arm, with per-arm rings capped at 5,000 samples (holdout.rs:192). unpaired_savings (holdout.rs:321):

reduction = 1 − mean_treatment / mean_control
Var(ratio) ≈ ratio² · (var_t/(n_t·mean_t²) + var_c/(n_c·mean_c²))     delta method
ci        = reduction ± 1.96 · sqrt(Var)

The mechanism for a negative net is structural, and it is worth understanding rather than explaining away. The treatment arm is the shaped arm: it runs reversible overflow, admitting budget / POINTER_TOKENS pointer-tier hits, plus compact serialisation. The control arm is forced unshaped: the legacy cumulative take_while drop plus pretty-printed JSON. Reversible overflow admits more candidates for the same budget; that is the recall win, so the shaped arm can emit more tokens than the arm that simply dropped them. When the recall cost exceeds the compaction saving, mean_treatment > mean_control and the reduction goes negative.

This is not a bug and it is not being hidden. There is a test asserting exactly this outcome, snapshot_separates_compaction_from_net (holdout.rs:473) pushes a treatment arm larger than its control, asserts snap.net.reduction < 0.0, and asserts a clean +25% on the compaction arm alongside it. The test's own comment reads "like the live finding".

A savings figure is meaningless without naming its arm. compaction is a cost feature and cannot be negative. net is the honest sum of a cost feature and a recall feature, and it goes negative whenever recall is winning. Quote the arm or do not quote the number.

Degenerate cases are guarded: an empty arm zeroes (holdout.rs:325), a non-positive control mean zeroes (holdout.rs:337), and fewer than two samples collapses the interval to the point estimate.

Units caveat. These numbers come from chars-over-four estimates of the emitted MCP payload (token_estimate.rs:27), not from a provider usage report, the opposite of the transcript cost lens in chapter 15, which uses real usage. Never quote the two as the same currency.

CRUX_BUDGET_REVERSIBLE has been removed

Reversible overflow shipped default-on in CO-3 (2026-06-25) and became unconditional in CO-5 (2026-06-30). The flag name survives only in comments (budget.rs:29, query.rs:587) and in a test comment. Setting it does nothing. The shaped/unshaped lever is now the holdout, not a flag.

CRUX_OUTPUT_HOLDOUT itself defaults to 0.0, which is off (holdout.rs:37). With fraction 0 no request is ever a control, sampling is a no-op, and token_savings returns the stub {"holdout_enabled": false} (token_savings.rs:33). The flag is not present in config.example.env. There is no live holdout measuring your daemon unless you turned one on.

11.13 Per-passport token accounting

Separate from savings, and unconditional. token_accounting.rs accumulates {calls, tokens_in, tokens_out, declared_budget_in} per passport in a process-local mutex (token_accounting.rs:41), bucketing unauthenticated callers under an ANON_PASSPORT sentinel (token_accounting.rs:19). It is deliberately not persisted to session state, to avoid journal spam (token_accounting.rs:16), and it resets on restart.

CORECRUXD_SESSION_TOKEN_BUDGET adds an optional limit and pct to the report; 0, empty or unset means no limit (token_accounting.rs:96). Nothing enforces it. session_token_usage is a reporting surface: no call is ever rejected or throttled because an accumulator crossed a threshold. The response self-labels "estimator": "chars/4" (token_usage.rs:71).

11.14 Status summary

CapabilityStatus
query_facts substring OR-match, decay ranking, budget trimSHIPPED
BM25 on the segment lane, global IDF, tenant-checkedSHIPPED
coverage.score / gaps[] (HTTP) / missing_tokens (MCP) / below_floorSHIPPED; the two surfaces differ in name and shape; token names are positional best-effort
Segment dense re-rank (0.7 BM25 + 0.3 cosine)SHIPPED, conditional on a provider factory and .ccxv companions; declares score_space per response
Default embedder LocalHashEmbedder, 256-dim feature hashingSHIPPED: lexical overlap, not meaning
Real embedding model (all-MiniLM-L6-v2, 384-dim)FLAG CORECRUXD_DENSE_MODEL=fastembed and a --features dense-embed-model build; the default build does not compile it
fused_retrieve three-lane fusionDECLARED-NOT-WIRED, test-only callers
Graph expansion in corecrux-retrieval::graphDECLARED-NOT-WIRED, no non-test caller
sparse fusion laneNot implemented, hardwired to 0.0 in the formula
token_budget on get_bootstrapNot a parameter, silently ignored if passed
query_expand returning document textNot present, metadata only in this build
CRUX_BUDGET_REVERSIBLERemoved, reversible overflow is unconditional
CRUX_OUTPUT_HOLDOUTFLAG, default 0.0 (off); token_savings is a stub until it is set
Per-passport token limit enforcementNot present, reporting only

Sources