The store platform
The CX Store is the platform's persistence spine: a **content-addressed object store** behind one document API. The same code runs against an in-memory store, a file on disk, an SQLite file, an S3 bucket, or a production daemon — you pick the substrate with a URL at open time and nothing else changes. It is specified in the store spec (`spec/03-approved/std-lib/store.md`); this section teaches the workflow. The per-function API reference is the `store` module page in the Standard library section of this guide, and the operator-depth documents live under `docs/dev/` in the repository (`store-embedded.md`, `store-service.md`, `store-management.md`, `store-security.md`).
Content addressing and identity
A document's ID **is** its content: the SHA-256 of its strict canonical bytes (see hash). That single decision buys the rest of the platform: identity is invariant across substrates, encodings, host bindings, and the wire, so replication is verifiable by construction and identical content stores exactly once — dedup is free, not a feature:
[?lib 'cx-stdlib/store' :as store]
[?let [= $s [$store:open 'mem://']]
[= $h [$store:put-doc $s [order [id 1] [amt 100]]]]
[= $again [$store:put-doc $s [order [id 1] [amt 100]]]]
[= $doc [$store:get-doc $s $h]]
([= $h $again], $doc)]
(true, [order [id 1] [amt 100]])
`get-doc` re-hashes after decode, so a corrupted or tampered object can never come back as a silently wrong document — an integrity mismatch is the hard error `CXER1120`. Identity is **two-tier** (the code-identity spec, `spec/03-approved/core/code-identity.md`): Tier-1 is the content hash of canonical bytes — the identity of *data* and the trust anchor for everything in this section; Tier-2 is code identity — `put-def`/`get-def` address CX *code* modulo renaming, so alpha-equivalent definitions share one key:
[?lib 'cx-stdlib/store' :as store]
[?let [= $s [$store:open 'mem://']]
[= $k1 [$store:put-def $s '[?def double ($x) [* $x 2]]']]
[= $k2 [$store:put-def $s '[?def double ($n) [* $n 2]]']]
([= $k1 $k2], [$store:get-def $s $k1])]
(true, '[?def double ($x) [* $x 2]]')
The substrate is chosen by the URL scheme at open time, and every effectful substrate is capability-gated deny-by-default (see capabilities):
| URL | substrate | grant needed |
|---|---|---|
| mem:// | in-process memory | none — capability-free |
| file:///path | local directory (pack format) | --allow-read / --allow-write |
| sqlite:///path.db | single SQLite file | --allow-read / --allow-write |
| s3://bucket/prefix | S3-compatible object storage | --allow-net |
| http(s):// · ftp:// · sftp:// | remote backends | --allow-net |
| cx-store+http(s)://host:port/name/ | the service daemon (next subsections) | --allow-net=host:port |
By default a store decomposes each document into a Merkle graph of shared subtree objects — identical subtrees across documents store once, a one-field edit re-stores only the path to the root, and `diff` skips identical subtrees in O(1). The alternative `document+` model (one object per doc, required for the columnar encodings) and the full URI axes are in the store spec and `docs/dev/store-embedded.md`.
The embedded workflow
`cx-stdlib/store` is the embedded tier — the store runs inside your process. Documents are immutable; "the latest X" lives in a separate **alias layer** that maps names to hashes, which is the one mutable surface. Updating means storing a new document (or deriving one with `modify-doc`, which applies any `[?modify]` action server-side-shaped and returns the new hash) and re-pointing the name:
[?lib 'cx-stdlib/store' :as store]
[?let [= $s [$store:open 'mem://']]
[= $v1 [$store:put-doc $s [config [retries 3]]]]
[= $v2 [$store:modify-doc $s $v1 [set-attr name=env value='prod']]]
[= $b [$store:branch $s 'main' $v2]]
([$store:get-doc $s [$store:get-alias $s 'main']],
[$store:get-doc $s $v1],
[$store:diff $s $v1 $v2])]
([config env=prod [retries 3]], [config [retries 3]], [diff [change path='/config' kind=modified]])
The original is untouched — history is free because nothing is ever overwritten. Across the whole corpus, `query` runs a CXPath selection over every held document:
[?lib 'cx-stdlib/store' :as store]
[?let [= $s [$store:open 'mem://']]
[= $h1 [$store:put-doc $s [invoice [total 120] [customer 'ada']]]]
[= $h2 [$store:put-doc $s [invoice [total 80] [customer 'grace']]]]
[$store:query $s '//total']]
([result hash=fa571812a0e85ea16e6d32b0b901a344bd2ac86cf78e18f2bb30b0d98c4b9037 ([total 120])], [result hash='0b129cf65cbf41d80663402f1d82d22e506c9f561815fadd64b8966c28165580' ([total 80])])
Going durable is changing the URL — the same program against `file://` persists, and a reopen finds the store self-describing (options are declared at create, not repeated at open):
[?lib 'cx-stdlib/store' :as store]
[?let [= $s [$store:open 'file:///tmp/notes-store']]
[$store:put-doc $s [note [text 'durable']]]]
$ cx notes.cx --allow-read --allow-write
Three habits worth forming from day one: open read paths read-only (`[$store:open-opts URL [map read-only='true']]` — then only the `read` grant is ever requested, and a write raises `CXER1110`); treat a Store handle as **single-owner** (sharing one across `[par]` workers raises `CXER1140` — open a handle per worker); and let denials teach you the grant — an ungranted open is a `CXER0271` error value naming the exact flag, so a program can degrade gracefully (see caps-errors). Streaming variants (`put-doc-stream`, `iter-docs`) keep memory bounded over any corpus size; the full surface is on the `store` module page.
Porcelain — introspect, branch, replicate
Over the content-addressed plumbing sits a deliberately git-shaped porcelain. Introspection and space reclamation:
[?lib 'cx-stdlib/store' :as store]
[?let [= $s [$store:open 'mem://']]
[= $h1 [$store:put-doc $s [doc [v 1]]]]
[= $h2 [$store:put-doc $s [doc [v 2]]]]
[= $del [$store:delete-doc $s $h1]]
[= $st [$store:status $s]]
[= $gcr [$store:gc $s]]
($st@docs, $gcr@reclaimed, [$count [$store:log $s]])]
(1, 4, 1)
| verb | what it does |
|---|---|
| status | held docs, distinct objects, dedup ratio, unflushed refs |
| log | the linear ref-log — each held doc-ref is one insertion-ordered epoch |
| prune / gc | reclaim objects no live ref reaches; gc adds durable compaction. A shared subtree survives the deletion of another doc — an object stays live while ANY ref reaches it |
| branch / branch-force | point a mutable named ref at a doc; a non-fast-forward move is refused (CXER1114, CAS-safe) unless forced |
| diff | structural diff by hash — identical subtrees are skipped, so cost is O(changed), not O(size) |
Replication is the same idea pointed at two stores. Because identity is content, transfer is incremental (only missing objects move), idempotent, and conflict-free — no CAS is needed on doc keys:
[?lib 'cx-stdlib/store' :as store]
[?let [= $laptop [$store:open 'mem://']]
[= $h [$store:put-doc $laptop [doc [item 'hello']]]]
[= $backup [$store:open 'mem://']]
[= $r [$store:push $laptop $backup]]
[$store:get-doc $backup $h]]
[doc [item 'hello']]
`push`/`pull`/`fetch` are the git-shaped incremental verbs; `clone` is an object-identity copy into an empty destination; `migrate` copies every doc and alias losslessly across **any** axis change (substrate, model, encoding) with every content-hash ID preserved. Backup is a `clone` to a second substrate; restore is a `clone` back, with every hash re-verifying on read; a substrate move (dev `mem://` to prod `s3://`, embedded to daemon) is a `migrate` — the recovery playbook is in `docs/dev/store-management.md`. `merge` and `rebase` deliberately do not exist: the model keeps roots plus epoch-ordered refs, not a parent-linked commit graph.
The service tier — cx store-serve
When the store must outlive one process or serve many, the **same** API, wire format, and content addressing move behind a purpose-built daemon: `cx store-serve` (also covered from the operator side in ops-store-daemon). It adds the operational layer — auth, RBAC, tenancy, observability, DoS fairness — and no storage smarts. The config is a CX document; one daemon mounts many named stores, each mount any embedded substrate URL:
[cxstore-service
[bind addr="127.0.0.1:18971"]
[stores [store name="docs" url="mem://docs"]]]
$ cx store-serve --config svc.cx --allow-net=127.0.0.1:18971
cx store-serve: listening on tcp://127.0.0.1:18971 — 1 store(s): docs (mem://docs), 4 workers, auth OPEN — no providers configured, anonymous full access (configure [auth …] for production)
Config validation is **attr-exact**: an unknown attribute is a startup error (`CXER1711`), never a silent drop. Liveness and readiness are plain HTTP, and the binary doubles as its own probe — `cx store-health` exits 0 iff the daemon reports accepting, which is what the Docker HEALTHCHECK and load balancers call:
$ curl -s http://127.0.0.1:18971/cx-store/v1/health
[health [status "ok"]]
$ curl -s http://127.0.0.1:18971/cx-store/v1/ready
[ready [accepting true] [draining false]]
$ cx store-health --url http://127.0.0.1:18971/cx-store/v1/ready && echo ready
ready
A client program is unchanged except for the URL — a `cx-store+http://` handle is a named remote store with the identical API (`cx-store+https://` for TLS):
[?lib 'cx-stdlib/store' :as store]
[?let [= $remote [$store:open 'cx-store+http://127.0.0.1:18971/docs/']]
[= $h [$store:put-doc $remote [doc [title 'over the wire']]]]
[$store:get-doc $remote $h]]
$ cx wire.cx --allow-net=127.0.0.1:18971
[doc [title 'over the wire']]
Auth is deny-by-default with four credential providers — static tokens (hashed at rest), JWT, DID (the agentic-principal path), and OIDC — all resolving to one principal shape with roles over the protocol's permission classes. `cx store-token` mints a bearer token and prints the ready-to-paste `[auth …]` stanza (the secret is shown once, never stored):
$ cx store-token --id ci --roles writer --tenant docs
[static [token id="ci" secret-hash="sha256:17af07a0…" roles="writer" tenant="docs"]]
cx store-token: secret for "ci" (shown ONCE, not stored — the config carries only the hash):
401cfe52…
Lifecycle: SIGTERM drains bounded and checkpoints every mounted store; SIGHUP hot-reloads the reloadable config (auth, limits, observability, timeouts, TLS contents) with validate-then-swap, all-or-nothing (`CXER1712` names any restart-required offender and refuses the whole reload). Prometheus metrics, optional OpenTelemetry tracing, structured request logs, per-principal rate/concurrency fairness, and the systemd/Docker deployment artifacts under `tooling/cxstore/` are covered in `docs/dev/store-service.md` and operations.
Transports — CSRP and gRPC
The daemon speaks two wire protocols over the same store surface — same ops, same semantics, same error codes:
| transport | client URL scheme | status |
|---|---|---|
| CSRP — HTTP/1.1 + CX bodies | cx-store+http:// (TLS: cx-store+https://) | the canonical, permanent interface (the CSRP spec, spec/03-approved/misc/cxstore-remote-protocol.md) |
| gRPC — HTTP/2 + protobuf | cx-store+grpc:// (TLS: cx-store+grpcs://) | opt-in second listener ([grpc enabled=true addr=…]) with normative CSRP parity; server-streaming for iter and large query (spec/03-approved/misc/cxstore-grpc.md) |
Only the scheme on `[$store:open]` changes — a client cannot tell the transports apart above the URL. Three runnable, self-contained recipes live in the repository under `examples/cxstore/`, each with a Makefile that starts the server, drives a client, and stops it (all three were run green against this branch-head binary):
- `client-server/` — CSRP client + server as two `cx` processes on loopback. The server is seven lines: open a store, listen, hand every accepted HTTP exchange to `[$store:csrp-handle]` — that one builtin is the whole server loop.
- `grpc/` — the same surface over gRPC: a daemon config exposing one store on **two** listeners (CSRP and gRPC), and a client driving put/get/list/query/modify through `cx-store+grpc://`.
- `dir-sync/` — a directory tree of `.cxd` files gains content addressing, dedup, history, and name-based lookup: `ingest.cx` walks and stores, `materialize.cx` writes the named tree back out, `watch.cx` keeps the store live off a real OS filesystem watch. Deliberately a recipe, not a subcommand — customization is editing CX.
Security and encryption at rest
Store security is four independent fail-closed layers — host capabilities at the effect point (capabilities), encryption at rest, service-tier RBAC/tenancy, and XAP-layer authority when a store backs a XAP (see dist-entitlements). The layer this subsection teaches is encryption: open a store with an `encrypt-key-id` and every object is sealed at rest (AES-256-CBC-then-HMAC, per-object data keys wrapped by a tenant key-encryption key). Crucially it is invisible to the object graph — objects stay keyed by the **plaintext** hash, so dedup, structural sharing, and replication are unchanged; only the bytes on the substrate are ciphertext:
[?lib 'cx-stdlib/store' :as store]
[?let [= $s [$store:open-opts 'file:///tmp/sealed-store' [map encrypt-key-id='tenant-a']]]
[= $h [$store:put-doc $s [secret-plan [step 'first']]]]
[$store:get-doc $s $h]]
$ CX_STORE_KEK_tenant-a=<64 hex chars> cx sealed.cx --allow-read --allow-write --allow-env
[secret-plan [step 'first']]
Key material never lives in config files — the KEK arrives by environment (`CX_STORE_KEK_<id>`). The mode is fixed at store creation and every mismatch is a hard error in both directions: an encrypted store opened without its key never appears empty or corrupt, encryption cannot be enabled in place on plaintext data, and a substrate that cannot seal refuses rather than degrading:
[?lib 'cx-stdlib/store' :as store]
[$store:open-opts 'mem://' [map encrypt-key-id='tenant-a']]
[err code=cx-err:CXER1100 message='E_STORE_UNRESOLVED_BACKEND: encryption-at-rest (encrypt-key-id) is supported on the sealing substrates — `file://…` (pack, the default), `file://…?encoding=object-per-key`, `sqlite://…`, or `s3://…` (got scheme=mem); refusing to store plaintext for mem://']
KEK rotation is a shipped operator verb: `cx store-rotate-kek` re-wraps every envelope's data key under a new tenant KEK — payloads and content addresses untouched, atomic per object, resumable, fail-closed (the same operation is callable in-process as `[$store:rotate-kek]`):
$ CX_STORE_KEK_tenant-a=… CX_STORE_KEK_tenant-b=… cx store-rotate-kek \
--url file:///tmp/sealed-store --encrypt-key-id tenant-a --new-key-id tenant-b
[rotation-report objects=5 rewrapped=5 already-current=0 from=tenant-a to=tenant-b]
cx store-rotate-kek: 5 object(s) now under `tenant-b` (5 re-wrapped) — verify reads, then destroy the old KEK `tenant-a`.
Service-tier RBAC and tenancy sit above this: the tenant boundary is **store-per-tenant**, so each tenant owns a separate dedup pool — a shared pool would let one tenant probe another's content by hash, and the design rejects that existence oracle by construction. The whole layer cake, the KMS seam, and the fail-closed inventory are in `docs/dev/store-security.md` and the security sections of the store and service-tier specs.
The management console
Fleet-facing management has a UI: the store management console, which lives in its **own repository** (`xap-store-console`) and is never embedded in the daemon. It is itself a XAP built from feature packages (the distribution system of the next section, eating its own dog food), connecting to daemons over CSRP with role-scoped credentials — so everything it can do is a public, RBAC-gated protocol op, and the console doubles as proof the management API is complete. The free tier covers connect, health, metrics, status, browse, config reload, and maintenance (gc). Contract: the store management console spec (`spec/03-approved/misc/store_management_console.md`); operator walkthrough: `docs/dev/store-management.md`. Bootstrap credentials with `cx store-token` (previous subsection).