CueCruxVaultCrux

SDKs · 4. Errors and retries

No Crux SDK has retry logic, backoff, jitter or an idempotency key. Every non-2xx becomes a thrown exception and the call ends there. That is a defensible design, retries belong in the caller, who knows whether the operation is safe to repeat, but it means the retry policy is your code, and getting it wrong on storeFact silently writes a second version of your fact.

This chapter is reference: the error classes, the status-by-status recoverability table, the two lists (safe, never), and a worked loop for each language.

4.1 The error class each client throws

ClientClassCarriesThrown when
@cuecrux/clientCoreCruxError extends Errorstatus: number, problem: ProblemDetails or nullAny non-2xx, except 404 on getFact, getSession, deleteFact (index.ts:36)
corecrux-clientCoreCruxError(Exception)status_code: int, detail: str, type: strAny status ≥ 400, except 404 on get_fact, get_session, delete_fact (errors.py:8)
@cuecrux/memoryCruxDaemonError extends Errorstatus: number, path: string, body: string or undefinedAny non-2xx, with no 404 convenience anywhere (errors.ts:2)
@cuecrux/engine-clientbare ErrorOnly a message, "<op> failed: <status>"Any non-OK response (index.ts:127)
FactoryClientFactoryClientError extends ErrorstatusCode, requestId, responseBody?Non-2xx after retries are exhausted (client.ts:48)

@cuecrux/engine-client is the outlier and the one to be careful with: its errors carry no status property. Recovering the status means parsing the message string, which is a contract nobody guaranteed. If you need to branch on status, wrap it in your own fetch and inspect the response there.

4.2 Not every failure is one of these classes

Three failure paths bypass the SDK error type entirely. They are the ones that break naive try / except CoreCruxError handling.

FailureWhat you actually catchWhere
Daemon unreachable, DNS failure, TLS failure, socket resetTypeError in TypeScript (the fetch rejection); httpx.ConnectError or httpx.ReadTimeout in PythonBelow the SDK, no CoreCruxError is constructed
A 200 with an unexpected body shapeKeyError in Python, from direct indexing of the response dict (client.py:59). TypeScript casts and does not notice_to_fact and its siblings
A 204 No ContentTypeScript returns undefined cast to the declared type (index.ts:265). Python returns {} (client.py:54)The request helper

In Python, catch httpx.HTTPError alongside CoreCruxError or transport failures escape your handler. In TypeScript there is no exported base class to catch, so guard with err instanceof CoreCruxError and re-throw everything else.

4.3 Status by status

R means the same request may succeed later without changing anything. N means it will not.

StatusMeaningRecoverableWhat to do
400Malformed for this route. Includes private: true on a fact write (facts.rs:229)NFix the payload
401No usable credential; "code": "UNAUTHENTICATED" (auth.rs:857)NSupply a token. See 3.3
403Identified and refused; "code": "MISSING_SCOPE" with missingScopes (auth.rs:1331)NGet a token carrying the named scopes
404No such fact, session or routeNBoth Crux SDKs already convert this to null or false on the three read-ish methods
409ConflictNRe-read, decide, write again. No SDK models this specially
422Semantic validation failureNFix the payload
429Quota exhausted. Carries Retry-After plus X-Crux-Quota-Limit and X-Crux-Quota-Remaining (quota.rs:23)RSleep for Retry-After, then retry once. FLAG: the middleware is inert unless CORECRUXD_QUOTA is on, default off (mod.rs:296)
500Unhandled daemon errorR, cautiouslyRetry a read. Do not retry a write without reading 4.5
502, 504A proxy in front of the daemon failedRRetry with backoff
503Daemon running, not ready. Body is {"ok": false, "checks": [...]} (health.rs:74)RBack off and retry. This is the normal startup and recovery signal

The 429 headers are the only backoff hint the daemon gives you, and they only exist when the operator enabled quota. Everywhere else you choose your own delay. Neither Crux SDK exposes response headers, so reading Retry-After requires your own fetch or httpx call, a real gap, and the reason the loops in 4.6 use a fixed schedule rather than the served value.

4.4 What is safe to retry

Everything in this list is a read. None of it mutates state, so repeating it costs latency and nothing else.

CallTypeScriptPython
Health and readinesshealthz, readyz, versionhealthz, readyz, version
Fact readsgetFact, getFactsByEntity, queryFacts, exportFactsget_fact, get_facts_by_entity, query_facts, export_facts
Session readgetSessionget_session
QueriestextSearch, textSearchExpand, graphExpand, timeRangetext_search, text_search_expand, graph_expand, time_range

Retry these on 429, 500, 502, 503, 504 and on transport failures. Cap the attempts, three is usually right, and back off exponentially with jitter so a fleet of agents does not synchronise against a recovering daemon.

exportFacts deserves a note: retrying a page is safe because the cursor is server-supplied and stable, but do not advance the cursor on a failed page. Retry the same cursor.

4.5 What must never be blindly retried

storeFact and storeFacts are not idempotent. This is the one that bites.

Writing the same (entity, key) a second time does not overwrite the first. The store assigns version = previous + 1, sets supersedes to the previous fact_id, and returns a new fact_id (fact_store.rs:882, fact_store.rs:1137). There is no idempotency key, no If-None-Match, and no client-supplied request id the daemon deduplicates on.

The failure mode is specific and quiet. Your write succeeds; the response is lost to a timeout or a reset connection; your retry lands; you now hold version 2 of a fact whose version 1 you never saw, and your recorded fact_id points at the wrong one. Nothing errors. Nothing warns you.

The safe sequence after an ambiguous write failure is read, then decide:

async function storeOnce(client: CoreCruxClient, fact: StoreFact): Promise<Fact> {
  try {
    return await client.storeFact(fact);
  } catch (err) {
    // The write may or may not have landed. Look before you leap.
    const { facts } = await client.getFactsByEntity(fact.entity);
    const existing = facts.find((f) => f.key === fact.key && f.value === fact.value && !f.deleted);
    if (existing) return existing;
    throw err;
  }
}

The rest of the never-retry-blindly list:

CallWhySafe alternative
putSession / put_sessionFull-document overwrite with no If-Match. A retry that races a concurrent writer silently discards their stateRead the current state, merge, write once. Accept that this is last-writer-wins
deleteFact / delete_factRepeating it is harmless, the tombstone is already written, but the return value flips: the second call 404s and yields falseTreat false as "not present", not as "failed"
Anything that failed with 400, 401, 403, 404, 409 or 422The response will be byte-identical next timeFix the request
FactoryClient.createJobJob creation. The SDK marks it retryUnsafe: false deliberately (client.ts:82), a retry would enqueue a duplicate jobPoll listJobs for your x-request-id before re-creating
@cuecrux/engine-client.answersA metered call against a hosted plane. A retry loop on a 5xx spends real budgetRetry at most once, and log it

The general rule: retry reads freely, retry writes only when you have re-read and know the write did not land.

4.6 Writing the loop yourself

TypeScript, wrapping any read method:

import { CoreCruxError } from "@cuecrux/client";

const RETRYABLE = new Set([429, 500, 502, 503, 504]);

export async function withRetry<T>(
  fn: () => Promise<T>,
  { attempts = 3, baseMs = 200, maxMs = 5_000 } = {},
): Promise<T> {
  let lastError: unknown;
  for (let attempt = 1; attempt <= attempts; attempt++) {
    try {
      return await fn();
    } catch (err) {
      lastError = err;
      const status = err instanceof CoreCruxError ? err.status : undefined;
      const transport = err instanceof TypeError;      // fetch rejection
      if (!transport && (status === undefined || !RETRYABLE.has(status))) throw err;
      if (attempt === attempts) break;
      const backoff = Math.min(maxMs, baseMs * 2 ** (attempt - 1));
      const jitter = Math.floor(Math.random() * (backoff / 4 + 1));
      await new Promise((r) => setTimeout(r, backoff + jitter));
    }
  }
  throw lastError;
}

// Reads only. Do not wrap storeFact in this.
const result = await withRetry(() => client.queryFacts({ query: "deploy", token_budget: 500 }));

Python, the same policy:

import random
import time

import httpx
from corecrux_client import CoreCruxError

RETRYABLE = {429, 500, 502, 503, 504}


def with_retry(fn, attempts: int = 3, base_ms: int = 200, max_ms: int = 5_000):
    last = None
    for attempt in range(1, attempts + 1):
        try:
            return fn()
        except CoreCruxError as exc:
            last = exc
            if exc.status_code not in RETRYABLE:
                raise
        except (httpx.ConnectError, httpx.ReadTimeout, httpx.RemoteProtocolError) as exc:
            last = exc
        if attempt == attempts:
            break
        backoff = min(max_ms, base_ms * 2 ** (attempt - 1))
        time.sleep((backoff + random.randint(0, backoff // 4)) / 1000)
    raise last


result = with_retry(lambda: client.query_facts("deploy", token_budget=500))

Both loops deliberately omit Retry-After. Neither SDK surfaces response headers, so the value the daemon served on a 429 is unreachable from inside the SDK. If quota enforcement matters to you, drop to a raw HTTP call for that path.

4.7 Timeouts

ClientTimeoutConfigurable
@cuecrux/clientNone. Global fetch with no signal (index.ts:242)No
corecrux-clientFlat 30 s, connect and read alike (client.py:154)Yes, at construction: timeout=
@cuecrux/memoryNone on requests; a 750 ms budget on the discovery probe only (discovery.ts:71)Probe only
FactoryClient10 s default, floor 1 s (client.ts:72)Yes, timeoutMs

The TypeScript client's lack of a timeout is the most likely cause of a hung agent. There is no AbortSignal parameter, so the only remedy is to race the promise yourself:

function withTimeout<T>(p: Promise<T>, ms: number): Promise<T> {
  return Promise.race([
    p,
    new Promise<never>((_, reject) =>
      setTimeout(() => reject(new Error(`timed out after ${ms}ms`)), ms),
    ),
  ]);
}

This rejects your promise but does not cancel the underlying request, the socket stays open until the platform closes it, and if the call was a write, it may still land. That is precisely the ambiguous-write case from 4.5.

4.8 The one SDK that does retry, and what it does

FactoryClient in SDKCrux is the only client in either estate with a retry policy, and it is worth copying rather than reinventing (client.ts:62):

ElementBehaviour
Retryable statusesExactly {500, 502, 503, 504} (client.ts:60). 429 is not in the set
Retryable transport errorsAbortError, FetchError, TypeError (client.ts:273)
AttemptsmaxRetries, default 2, floor 0
Backoffmin(max, base × 2^(n−1)) plus jitter of up to a quarter of that (client.ts:289)
DefaultsretryBaseDelayMs 150 (floor 50), retryMaxDelayMs 2000
Per-method opt-inGETs retry; the one POST that creates a job does not (client.ts:82)
CorrelationEvery request carries x-request-id, caller-supplied or a fresh randomUUID() (client.ts:243)
Response validationEvery body is schema.safeParsed before it is returned

The per-method opt-in is the design decision worth stealing: retryability is a property of the operation, not of the status code alone.

SDKCrux also ships a status-to-taxonomy mapper in its shared runtime, mapHttpError, with an explicit isRecoverable predicate (errors.ts:11):

type SDKError =
  | { type: 'RateLimited'; message?: string; retryAfterSec?: number }
  | { type: 'UpstreamDown'; message?: string; upstream?: string }
  | { type: 'ValidationError'; message?: string; issues?: ZodIssue[] }
  | { type: 'Contradiction'; message?: string; details?: string }
  | { type: 'StaleSnapshot'; message?: string; asOf?: string; observedAt?: string }
  | { type: 'AuthMissing'; message?: string };

// isRecoverable → true for RateLimited, UpstreamDown, StaleSnapshot
//                 false for ValidationError, Contradiction, AuthMissing
// mapHttpError: 429→RateLimited (+retryAfterSec), 401/403→AuthMissing, 409→Contradiction,
//               428→StaleSnapshot, 422→ValidationError, ≥500→UpstreamDown

That taxonomy lives in @cuecrux-internal/core, which is a restricted package on GitHub Packages - you cannot install it from public npm. It is reproduced here because the classification is sound and you may want it in your own code, not because you can import it.

4.9 The 3am checklist

SymptomFirst checkLikely cause
Every call hangs, no errorIs this TypeScript?No timeout exists. Add one (4.7)
CoreCruxError: 401 on every callCORECRUXD_AUTH_MODE on the daemonUnder dev_scopes the token must be a scope list, not a JWT
CoreCruxError: 403 with no obvious causeThe missingScopes array in the problem bodyToken is missing facts:write or query:read. Python discards this field, use curl
Everything 503s after a restartcurl /readyz and read checksA readiness gate. Often disk capacity on the data partition
Duplicate facts appearing under one keyYour retry policystoreFact is not idempotent (4.5)
KeyError from a Python call that "worked"The daemon version_to_fact indexes required keys directly; a shape change surfaces as KeyError, not CoreCruxError
SSE connects then immediately errors, no statusWhether the daemon requires authEventSource cannot send the token (3.9)
TypeError: fetch failedThe base URL and the daemon processTransport failure, not an API error. Not a CoreCruxError

Sources