The store platform

Content addressing and identity

A document's ID is its content: the SHA-256 of its strict canonical bytes (see hash). That single decision buys the rest of the platform: identity is invariant across substrates, encodings, host bindings, and the wire, so replication is verifiable by construction and identical content stores exactly once — dedup is free, not a feature:

      [?lib 'cx-platform/store' as=store]
[?let [= $s [$store:open 'mem://']]
      [= $h [$store:put-doc $s [order [id 1] [amt 100]]]]
      [= $again [$store:put-doc $s [order [id 1] [amt 100]]]]
      [= $doc [$store:get-doc $s $h]]
  ([= $h $again], $doc)]
    

      (true, [order [id 1] [amt 100]])
    

get-doc re-hashes after decode, so a corrupted object can never come back as a silently wrong document — an integrity mismatch is the hard error CXER1120. Documents split by identity rule: a structured document (cx data) is keyed by the hash of its canonical bytes, and the canonicalized round-trip above is the contract; an opaque document — CX code, images, plain text, anything whose bytes ARE the content — is keyed by the hash of its raw bytes via put-blob, and get-blob returns those bytes exactly. Code's "same function?" question is answered separately by the pure [$cx:computation-id] claim — an index, never a storage key — so alpha-equivalent definitions share one claim while each spelling keeps its own byte-exact address:

      [?lib 'cx-platform/store' as=store]
[?let [= $s [$store:open 'mem://']]
      [= $src '[?def double ($x) [* $x 2]]']
      [= $back [$store:get-blob $s [$store:put-blob $s $src]]]
  ([= $back $src],
   [= [$cx:computation-id '[?def double ($x) [* $x 2]]']
      [$cx:computation-id '[?def double ($n) [* $n 2]]']])]
    

      (true, true)
    

The substrate is chosen by the URL scheme at open time, and every effectful substrate is capability-gated deny-by-default (see capabilities):

By default a store decomposes each document into a Merkle graph of shared subtree objects — identical subtrees across documents store once, a one-field edit re-stores only the path to the root, and diff skips identical subtrees in O(1). The alternative document+ model (one object per doc, the required prefix for the columnar encodings) is the other axis of the same URL — [document+]<substrate>://<location>[?encoding=…&read-only=…&schema=…] — and we ask you to state only what departs from a default, only at create: a file:// store is self-describing on reopen. An axis we have specified but not applied (a non-none ?compression=, the ?cache= layer) is refused fail-closed, never accepted and ignored, and a scheme we do not build dispatches to a refusal that names it:

        [?lib 'cx-platform/store']
[$store:open "gopher://h/dir/"]
      

        $ cx --allow-net prog.cx
[err code=cx-err:CXER1100 message='E_STORE_UNRESOLVED_BACKEND: unknown URL scheme in gopher://h/dir/' at='2:1']
      

The embedded workflow

cx-platform/store is the embedded tier — the store runs inside your process. Documents are immutable; "the latest X" lives in a separate alias layer that maps names to hashes, which is the one mutable surface. Updating means storing a new document (or deriving one with modify-doc, which applies any [?modify] action server-side-shaped and returns the new hash) and re-pointing the name:

      [?lib 'cx-platform/store' as=store]
[?let [= $s [$store:open 'mem://']]
      [= $v1 [$store:put-doc $s [config [retries 3]]]]
      [= $v2 [$store:modify-doc $s $v1 [set-attr name=env value='prod']]]
      [= $b [$store:branch $s 'main' $v2]]
  ([$store:get-doc $s [$store:get-alias $s 'main']],
   [$store:get-doc $s $v1],
   [$store:diff $s $v1 $v2])]
    

      ([config env=prod [retries 3]], [config [retries 3]], [diff [change path='/config' kind=modified]])
    

The original is untouched — history is free because nothing is ever overwritten. Across the whole corpus, query runs a CXPath selection over every held document:

      [?lib 'cx-platform/store' as=store]
[?let [= $s [$store:open 'mem://']]
      [= $h1 [$store:put-doc $s [invoice [total 120] [customer 'ada']]]]
      [= $h2 [$store:put-doc $s [invoice [total 80] [customer 'grace']]]]
  [$store:query $s '//total']]
    

      ([result doc=sha2-256:e5b689fa494377cea041d4bf6496e28e97dd13a059aa4c22c75360fc7d0ceb30 source='mem://#//total' [total 120]], [result doc='sha2-256:89123d4330eda07c2dbeb3878cd1ca12760c1d19e37fdb634b4737bfc2784924' source='mem://#//total' [total 80]])
    

Going durable is changing the URL — the same program against file:// persists, and a reopen finds the store self-describing (options are declared at create, not repeated at open):

      [?lib 'cx-platform/store' as=store]
[?let [= $s [$store:open 'file:///tmp/notes-store']]
  [$store:put-doc $s [note [text 'durable']]]]
    

      $ cx --allow-read --allow-write notes.cx
    

Three habits worth forming from day one: open read paths read-only ([$store:open-opts URL [map read-only='true']] — then only the read grant is ever requested, and a write raises CXER1110); treat a Store handle as single-owner (sharing one across [par] workers raises CXER1140 — open a handle per worker); and let denials teach you the grant — an ungranted open is a CXER0271 error value naming the exact flag, so a program can degrade gracefully (see caps-errors). Streaming variants (put-doc-stream, iter-docs) keep memory bounded over any corpus size; the full surface is on the store module page. The read-only refusal and the ungranted open, replayed:

        [?lib 'cx-platform/store']
[?let [= $s [$store:open-opts "mem://" [map read-only="true"]]] [$store:put-doc $s [doc [item "x"]]]]
      

        $ cx prog.cx
[err code=cx-err:CXER1110 message='E_STORE_READ_ONLY: mem://' at='2:65']
      

        [?lib 'cx-platform/store']
[$store:open "file:///var/data/store/"]
      

        $ cx prog.cx
[err code=cx-err:CXER0271 message='E_CAP_DENIED: write capability required for store open file:///var/data/store/; none granted (grant via --allow-write)' at='2:1']
      

One writer per local root. A file://, sqlite:// or local columnar root admits ONE writable handle plus any number of read-only ones — across processes as much as within one. A second writable open in the same process returns a handle over the same live store; a second writable open from another process is refused CXER1143, naming the holding pid. Two writers on one root corrupt it, and the damage only shows at a later open — so when you need many writers, run the service tier (one writer, N clients) or open read-only. A crashed writer releases the root at once: the lock dies with the process, and nothing is cleaned up by hand (the store spec's locking section). Streaming and direct storage agree byte for byte, and an opaque blob comes back exactly as it went in:

        [?lib 'cx-platform/store']
[?let [= $s [$store:open "mem://"]] [= $a [$store:put-doc $s [user name=al]]] [= $b [$store:put-doc-stream $s [user name=al]]] [= $a $b]]
      

        $ cx prog.cx
true
      

        [?lib 'cx-platform/store']
[?let [= $s [$store:open "mem://"]] [= $src "[?def greet ($name) [str-format 'hello, {}' $name]]"] [$eq [$store:get-blob $s [$store:put-blob $s $src]] $src]]
      

        $ cx prog.cx
true
      

The errors you meet first, each a value naming its cause (the store spec's error-codes section carries the table):

Porcelain — introspect, branch, replicate

Over the content-addressed plumbing sits a deliberately git-shaped porcelain. Introspection and space reclamation:

      [?lib 'cx-platform/store' as=store]
[?let [= $s [$store:open 'mem://']]
      [= $h1 [$store:put-doc $s [doc [v 1]]]]
      [= $h2 [$store:put-doc $s [doc [v 2]]]]
      [= $del [$store:delete-doc $s $h1]]
      [= $st [$store:status $s]]
      [= $gcr [$store:gc $s]]
  ($st@docs, $gcr@reclaimed, [$count [$store:log $s]])]
    

      (1, 4, 3)
    

Each verb, replayed: a delete is an epoch in the log, a branch advance appears in it, gc reclaims only what no ref reaches, a non-fast-forward branch move is refused, and a one-leaf change diffs to one path:

        [?lib 'cx-platform/store']
[?let [= $s [$store:open "mem://"]]
  [= $h1 [$store:put-doc $s [doc [v 1]]]]
  [= $h2 [$store:put-doc $s [doc [v 2]]]]
  [= $b1 [$store:branch $s "main" $h1]]
  [= $b2 [$store:branch-force $s "main" $h2]]
  [= $l [$store:log $s]]
  [?for [in $r $l//ref[= $_@plane "aliases"]]
    [yield [$text $r/@pos]]]]
      

        $ cx prog.cx
'1'
'2'
      

        [?lib 'cx-platform/store']
[?let [= $s [$store:open "mem://"]] [= $a [$store:put-doc $s [doc [item "x"]]]] [= $b [$store:put-doc $s [doc [item "y"]]]] [= $d [$store:delete-doc $s $a]] [$store:gc $s]]
      

        $ cx prog.cx
[gc-result reclaimed=2 objects=2]
      

        [?lib 'cx-platform/store']
[?let [= $s [$store:open "mem://"]] [= $h1 [$store:put-doc $s [doc [v 1]]]] [= $h2 [$store:put-doc $s [doc [v 2]]]] [= $b1 [$store:branch $s "main" $h1]]
  [$store:branch $s "main" $h2]]
      

        $ cx prog.cx
[err code=cx-err:CXER1114 message='E_STORE_REF_CONFLICT: branch main points at sha2-256:5843b94b7af592c434acd9d8816e45f5bc50ebce67ea94a0bcb206202ccaeaed, not sha2-256:34bb7bd03048150a5f9b3fc52fb264f378d71d6f03a1ac0b0bfbdab43d1dac35 — use branch-force to move it' at='3:3']
      

        [?lib 'cx-platform/store']
[?let [= $s [$store:open "mem://"]] [= $a [$store:put-doc $s [order [id 1] [amt 100]]]] [= $b [$store:put-doc $s [order [id 1] [amt 200]]]] [$store:diff $s $a $b]]
      

        $ cx prog.cx
[diff [change path='/order/amt/[0]' kind=modified]]
      

Replication is the same idea pointed at two stores. Because identity is content, transfer is incremental (only missing objects move), idempotent, and conflict-free — no CAS is needed on doc keys:

      [?lib 'cx-platform/store' as=store]
[?let [= $laptop [$store:open 'mem://']]
      [= $h [$store:put-doc $laptop [doc [item 'hello']]]]
      [= $backup [$store:open 'mem://']]
      [= $r [$store:push $laptop $backup]]
  [$store:get-doc $backup $h]]
    

      [doc [item 'hello']]
    

push/pull/fetch are the git-shaped incremental verbs; clone is an object-identity copy into an empty destination; migrate copies every doc and alias losslessly across any axis change (substrate, model, encoding) with every content-hash ID preserved. Backup is a clone to a second substrate; restore is a clone back, with every hash re-verifying on read; a substrate move (dev mem:// to prod s3://, embedded to daemon) is a migrate. merge and rebase deliberately do not exist: the model keeps roots plus epoch-ordered refs, not a parent-linked commit graph. The three verbs, replayed — a clone into an empty destination, a push that carries only what the other side lacks, and a migration's report:

        [?lib 'cx-platform/store']
[?let [= $a [$store:open "mem://"]] [= $h [$store:put-doc $a [doc [item "hello"]]]] [= $b [$store:open "mem://"]] [= $r [$store:clone $a $b]] [$store:get-doc $b $h]]
      

        $ cx prog.cx
[doc [item hello]]
      

        [?lib 'cx-platform/store']
[?let [= $l [$store:open "mem://"]] [= $h [$store:put-doc $l [doc [v 1]]]] [= $rm [$store:open "mem://"]] [= $r [$store:push $l $rm]] [$store:get-doc $rm $h]]
      

        $ cx prog.cx
[doc [v 1]]
      

        [?lib 'cx-platform/store']
[?let [= $a [$store:open "mem://"]] [= $h [$store:put-doc $a [doc [item "hello"]]]] [= $b [$store:open "mem://"]] [$store:migrate $a $b]]
      

        $ cx prog.cx
[migration-report doc-count=1 hashes-verified=1 bytes-written=18]
      

The recovery playbook we run ourselves, in four lines:

• Backup is a clone (or a push) to a second substrate — nightly file:// to s3:// — incremental and verifiable by construction, because identity is content.
• Restore is a clone back into a fresh store; every hash re-verifies on read.
• A substrate move (file to sqlite, dev mem:// to prod s3://, embedded to daemon) is a migrate; document identity is universal, so nothing that references a hash changes.
• Crash safety inside one store is the engine's append-only ref-log recovery plus a per-op atomic-rename flush; a failed persist raises CXER1116 rather than acknowledging a phantom write, and the next successful persist self-heals.

Object-identity transfer is subtree to subtree only; across the model boundary the verbs degrade to the doc-level migrate. On a service handle (cx-store://, next subsection) status and gc are the daemon's admin-plane ops under the same names, scoped to the mount the session attached to; mounts enumerates the daemon's stores and config-reload triggers the validate-then-swap reload. All four sit in the admin class of the one grant table, and all four are daemon-level — a local handle IS its only store, so there they refuse with CXER1709:

        [?lib 'cx-platform/store']
[?let [= $s [$store:open "mem://"]] [$store:mounts $s]]
      

        $ cx prog.cx
[err code=cx-err:CXER5012 message='E_XSP_STORE_UNSUPPORTED: mounts is a daemon-level op — it requires a cx-store:// service-tier handle' at='2:37']
      

        [?lib 'cx-platform/store']
[?let [= $s [$store:open "mem://"]] [$store:config-reload $s]]
      

        $ cx prog.cx
[err code=cx-err:CXER5012 message='E_XSP_STORE_UNSUPPORTED: config-reload is a daemon-level op — it requires a cx-store:// service-tier handle' at='2:37']
      

The service tier — cx store-serve

When the store must outlive one process or serve many, the same API, wire format, and content addressing move behind a purpose-built daemon: cx store-serve (also covered from the operator side in ops-store-daemon). It adds the operational layer — XSP-AUTH grant-table authority, tenancy, observability, DoS fairness — and no storage smarts. The config is a CX document; one daemon mounts many named stores, each mount any embedded substrate URL:

      [cxstore-service
  [bind addr="127.0.0.1:18971"]
  [stores [store name="docs" url="mem://docs"]]]
    

      $ cx store-serve --config svc.cx --allow-net=127.0.0.1:18971
    

      cx store-serve: listening on tcp://127.0.0.1:18971 — 1 store(s): docs (mem://docs), 4 workers, auth OPEN — no [xsp [grants …]] configured, anonymous full access (configure grants for production)
    

Config validation is attr-exact: an unknown attribute is a startup error (CXER1711), never a silent drop. Liveness and readiness are plain HTTP, and the binary doubles as its own probe — cx store-health exits 0 iff the daemon reports accepting, which is what the Docker HEALTHCHECK and load balancers call:

      $ curl -s http://127.0.0.1:18971/cx-store/v1/health
[health [status "ok"]]
$ curl -s http://127.0.0.1:18971/cx-store/v1/ready
[ready [accepting true] [draining false]]
$ cx store-health --url http://127.0.0.1:18971/cx-store/v1/ready && echo ready
ready
    

A client program is unchanged except for the URL — a cx-store:// handle (the XSP store profile — THE store wire; cx-store+xsp:// is the cleartext-dev sibling) is a named remote store with the identical API:

      [?lib 'cx-platform/store' as=store]
[?let [= $remote [$store:open 'cx-store+xsp://127.0.0.1:18971/docs/']]
      [= $h [$store:put-doc $remote [doc [title 'over the wire']]]]
  [$store:get-doc $remote $h]]
    

      $ cx --allow-net=127.0.0.1:18971 wire.cx
[doc [title 'over the wire']]
    

Authority is XSP-AUTH, deny-by-default: a client presents a verifiable credential (a DID it holds, plus an optional delegation chain), and the daemon's [xsp [grants …]] table — principal-rooted delegations over the capability classes read/write/delete/admin — decides per op. There is ONE authority calculus; the retired bearer/RBAC plane and the [auth …] config section are gone. A client names its identity through open-opts (xsp-did + xsp-seed-env); the seed never rides a URL.

Lifecycle: SIGTERM drains bounded and checkpoints every mounted store; SIGHUP hot-reloads the reloadable config sections (tls, timeouts, limits, observability) with validate-then-swap, all-or-nothing (CXER1712 names any restart-required offender and refuses the whole reload). Two rules the URL enforces rather than documents: the port is required on cx-store(+xsp):// — the profile listener has no registered default, so a portless URL refuses at open rather than dialing something arbitrary — and the URL carries no userinfo (a user@host authority refuses with CXER1100; identity rides the open-opts, the seed always from the environment). The retired cx-store+http(s):// schemes refuse with CXER1100 naming the replacements — a cutover, no dual-accept window. And like every remote substrate, a service open is capability-gated:

        [?lib 'cx-platform/store']
[$store:open "cx-store+xsp://127.0.0.1:9099/teststore/"]
      

        $ cx prog.cx
[err code=cx-err:CXER0271 message='E_CAP_DENIED: net capability required for store open cx-store+xsp://127.0.0.1:9099/teststore/; none granted (grant via --allow-net)' at='2:1']
      

Bootstrapping a deployment from a clean state (mint, grant, serve, present), the one authority calculus, metrics and tracing, DoS fairness, the reload's hot and restart-required sections, and the systemd/Docker artifacts under tooling/cxstore/ are the operator's page: ops-bootstrap, ops-authority, ops-observability, ops-artifacts.

Transports — the store wire and the edge

The daemon serves the same store surface over two wires — same ops, same semantics, same error codes:

Only the scheme on [$store:open] changes — a client cannot tell the transports apart above the URL. Three runnable, self-contained recipes live in the repository under examples/cxstore/, each with a Makefile that starts the server, drives a client, and stops it (all three were run, and passed, against this branch-head binary):

• client-server/ — a client over THE store wire: cx store-serve holds a store and serves it over the XSP profile, and a client drives put/get/list/query through cx-store+xsp:// — remoting is a URL change, not an API change.
• grpc/ — the same surface over the integration edge: a daemon config exposing one store on a gRPC listener, and a client driving put/get/list/query/modify through cx-store+grpc://.
• dir-sync/ — a directory tree of .cxd files gains content addressing, dedup, history, and name-based lookup: ingest.cx walks and stores, materialize.cx writes the named tree back out, watch.cx keeps the store live off a real OS filesystem watch. Deliberately a recipe, not a subcommand — customization is editing CX.

Security and encryption at rest

Store security is four independent fail-closed layers — host capabilities at the effect point (capabilities), encryption at rest, service-tier grants/tenancy, and XAP-layer authority when a store backs a XAP (see dist-entitlements). The layer this subsection teaches is encryption: open a store with an encrypt-key-id and every object is sealed at rest (AES-256-CBC-then-HMAC, per-object data keys wrapped by a tenant key-encryption key). Crucially it is invisible to the object graph — objects stay keyed by the plaintext hash, so dedup, structural sharing, and replication are unchanged; only the bytes on the substrate are ciphertext:

      [?lib 'cx-platform/store' as=store]
[?let [= $s [$store:open-opts 'file:///tmp/sealed-store' [map encrypt-key-id='tenant-a']]]
      [= $h [$store:put-doc $s [secret-plan [step 'first']]]]
  [$store:get-doc $s $h]]
    

      $ CX_STORE_KEK_tenant-a=<64 hex chars> cx --allow-read --allow-write --allow-env sealed.cx
[secret-plan [step 'first']]
    

Key material never lives in config files — the KEK arrives by environment (CX_STORE_KEK_<id>). The mode is fixed at store creation and every mismatch is a hard error in both directions: an encrypted store opened without its key never appears empty or corrupt, encryption cannot be enabled in place on plaintext data, and a substrate that cannot seal refuses rather than degrading:

      [?lib 'cx-platform/store' as=store]
[$store:open-opts 'mem://' [map encrypt-key-id='tenant-a']]
    

      [err code=cx-err:CXER1100 message='E_STORE_UNRESOLVED_BACKEND: encryption-at-rest (encrypt-key-id) is supported on the sealing substrates — `file://…` (pack, the default), `file://…?encoding=object-per-key`, `sqlite://…`, or `s3://…` (got scheme=mem); refusing to store plaintext for mem://']
    

KEK rotation is a shipped operator verb: cx store-rotate-kek re-wraps every envelope's data key under a new tenant KEK — payloads and content addresses untouched, atomic per object, resumable, fail-closed (the same operation is callable in-process as [$store:rotate-kek]) — and on a store that was never sealed there is nothing to re-wrap, so the verb refuses rather than pretending:

        [?lib 'cx-platform/store']
[?let [= $s [$store:open "mem://"]] [$store:rotate-kek $s "tenant-b"]]
      

        $ cx prog.cx
[err code=cx-err:CXER1141 message='E_STORE_ROTATION_UNSUPPORTED: mem:// is not encrypted at rest — rotate-kek re-wraps envelope data keys and has nothing to rotate on a plaintext store (store.md §9.1)' at='2:37']
      

The shipped rotation, end to end:

      $ CX_STORE_KEK_tenant-a=… CX_STORE_KEK_tenant-b=… cx store-rotate-kek \
    --url file:///tmp/sealed-store --encrypt-key-id tenant-a --new-key-id tenant-b
[rotation-report objects=5 rewrapped=5 already-current=0 from=tenant-a to=tenant-b]
cx store-rotate-kek: 5 object(s) now under `tenant-b` (5 re-wrapped) — verify reads, then destroy the old KEK `tenant-a`.
    

Service-tier grant authority and tenancy sit above this: the tenant boundary is store-per-tenant, so each tenant owns a separate dedup pool — a shared pool would let one tenant probe another's content by hash, and the design rejects that existence oracle by construction. Scoping rides the authority basis rather than a principal-to-store-name list: a grant, and any delegation attenuated from it, is bound to its mount, and a cross-tenant delegation is a fault, never something that quietly compiles to nothing. A class the row does not hold is denied as a value — [deny …] carrying its code as data, which is what lets the daemon send the same value across the wire verbatim:

        [?lib 'cx-stdlib/authz']
[?lib 'cx-platform/authz-store']
[?let [= $az [$authz-store:store {tenant: 'docs'}]]
  [= $g [$authz-store:delegate $az
    [delegation grant-1 [tenant docs]
      [from [principal 'did:key:z6Mkgxj2R3HLtQRpPnvfvpuKEceSqf3tZHBjdmZ3fFz3JHGG']]
      [to [agent 'did:key:z6MkneMkZqwqRiU5mJzSG3kDwzt9P8C59N4NGTfBLfSGE7c7']]
      [capabilities [read]] [over '/orders'] [assurance :t1] [signature sig-daemon]]]]
  [$authz:check $az [authz-request
    [actor [agent 'did:key:z6MkneMkZqwqRiU5mJzSG3kDwzt9P8C59N4NGTfBLfSGE7c7']]
    [capability write] [slice '/orders/9'] [tenant docs]]]]
      

        $ cx prog.cx
[deny actor=did:key:z6MkneMkZqwqRiU5mJzSG3kDwzt9P8C59N4NGTfBLfSGE7c7 [code cx-err:CXER4700] [reason :no-grant] [capability write] [slice /orders/9] [tenant id=docs]]
      

Three least-privilege habits: open registries and read paths with [map read-only='true'] so write is never requested; scope net grants to the daemon (--allow-net=127.0.0.1:18762), never bare --allow-net; and keep --allow-all for trusted local runs. A principal's seed lives in a 0600 file and reaches the process through xsp-seed-env — never a URL, an opts literal or the config text; ${env:VAR} injection keeps TLS key material out of the config file; the request log carries no credential material. When a store backs a XAP, a fourth layer sits above these: grants attach to features and principals at the policy enforcement point, never to library code, install-time consent is exactly the manifest's needs set, and an entitlement check is a policy check (dist-entitlements). What refuses rather than degrades, in one table:

The management console

Fleet-facing management has a UI: the store management console, which lives in its own repository (xap-store-console) and is never embedded in the daemon. It is itself a XAP built from feature packages (the distribution system of the next section, eating its own dog food), connecting to daemons over the XSP store profile with a DID-held credential — so everything it can do is a public, grant-gated protocol op, and the console doubles as proof the management API is complete. The free tier covers connect, health, metrics, status, browse, config reload, and maintenance (gc). Contract: the store management console spec (spec/03-approved/misc/store_management_console.md), read through the wire-transition note at its head, which maps every store HTTP op verb-for-verb onto the profile. Client identity is a DID presented over the wire (open-opts xsp-did/xsp-seed-env); operator authority rides [xsp [grants …]]. The console's own principal is minted the way every principal is — cx store-mint-principal, offline, granting nothing until an operator installs the stanza (ops-bootstrap) — and gets the narrowest classes its tier needs: read covers browsing (get, list, iter, query); the whole admin plane — status, gc, mounts, config-reload — sits in admin.