Concepts
Homoiconicity
CX is homoiconic: code is data, in a strict sense. The reader, the AST, the canonical bytes, and the wire format all treat a program as a CX document. There is no separate syntax tree for code versus data — a `[?for]` directive is an element whose name happens to start with `?`, with attributes and a body and a closing bracket, identical in structure to `[user]` or `[order]`.
The practical consequences fall out of the equality "program = document." Every tool that consumes data also consumes code, and every operation that transforms data also transforms code. There is no impedance mismatch at the seam — because there is no seam.
- **Code can be inspected with CXPath.** A program is a tree; you can search it with `//directive[@name='?for']`, count its directives with `count(//*[starts-with(name(), '?')])`, find every CXPath literal embedded inside it, extract every `[yield …]` clause body, or hash it for cache keys. The same CXPath vocabulary you use to query a JSON-shape user database queries the program that processes that database.
- **Macros are unnecessary.** A directive is data; you construct it with the same element-construction syntax you use for data. Whatever a macro system would let you do (generate code, splice quoted fragments, abstract over patterns) you do by building or transforming AST elements directly. The `[?modify]` primitive (see modify) doubles as a code transformer when the focused expression happens to be a directive.
- **One toolchain.** Parser, lexer, formatter, validator, linter, LSP, tree-sitter — one each, used identically on code and data. The grammar in [`spec/grammar.ebnf`](../../spec/grammar.ebnf) describes both. The tree-sitter grammar ([`tooling/tree-sitter-cx`](../../tooling/tree-sitter-cx)) parses both. `cx fmt` formats both. `cx lint` checks both. Editor extensions highlight both with the same rules.
- **One identity contract.** A program has a SHA-256 just like a document does. Caching, content-addressing, audit trails, and reproducible test fixtures all work the same way for code as for data. The build cache for a CX pipeline keys on the SHA-256 of the canonicalized program text; if the program is byte-equal to a previous run, the cache hits even when the textual layout differs.
- **One projection table.** Every CX value — code or data — projects to JSON, YAML, TOML, XML, Markdown, and data-bin. A `[?for]` directive in JSON is just an object with `_cx_kind: "directive"` and the same attribute and body slots. Programs round-trip through every projection the data language does, which is what makes "ship code alongside data over the wire" trivial.
# A program IS a document — the same tools apply unchanged.
$ cat prog.cx
[?for [in $x (1, 2)] [yield [n $x]]]
# Examine the program as data: its AST is ordinary JSON.
$ cx --ast prog.cx
{"type":"Document","elements":[{"type":"EvalDirective","name":"for", …}]}
# Identity works on code exactly as on data — a cache key:
$ cx hash prog.cx
4c3aec212e8174d1…
# Reformat-stable: cx fmt output hashes identically.
$ cx fmt prog.cx | cx hash /dev/stdin
**The reader is the AST builder.** Where most languages have a tokenizer, then a parser that consumes tokens to build an AST, CX has one phase. The reader emits AST nodes directly; the "tokens" that flow through it are the same `Element` values the rest of the system manipulates. This is what makes self-modifying code, AST-level macros, and program-as-data affordances cheap — there is no extra layer to lift across.
**Two practical patterns that exploit homoiconicity.** First, **deploy-time program rewrite**: a build script loads the deployment manifest as data, finds every `[?http-client]` directive whose `:base-url` is the staging host, and rewrites to production. The rewrite is `[?modify]` with a path predicate — no string substitution, no template engine. Second, **inspect-and-cache**: a CXPath compiles to a tree of step nodes; if you've already evaluated a CXPath against a document, the cache key is the hash of the path tree, and the cache survives reformatting (`/users[1]` and `/users [ 1 ]` hash identically because canonical form collapses incidental whitespace).
Errors as values
An error in CX is an ordinary value — a CX element with a reserved error kind. Errors flow through expressions as first-class data, get pattern-matched in `[?match]`, get propagated by `?` and `!`, and get recovered by `[?match]`, `[?else]`, and `[?fallback]`. There is no parallel exception stack, no stack-unwinding control flow, no try/catch ladder distinct from the expression tree. An error is a value; an error-producing expression is an expression; the type rules treat them uniformly.
**The error namespace.** CX error codes are partitioned by layer. Each range is reserved per-spec; the full table is in [`spec/code.md`](../../spec/code.md) §9.4.
| range | owner |
|---|---|
| CXER0000–CXER0099 | Parser / lexer |
| CXER0100–CXER0299 | CX code language (eval) |
| CXER0300–CXER0399 | Schema validator |
| CXER0400–CXER0499 | Conversion / projection |
| CXER0500–CXER0599 | Services + clients |
| CXER0600–CXER0699 | Workers + channels |
| CXER0700–CXER0799 | Async + cancellation |
| CXER0800–CXER0899 | I/O + filesystem |
| CXER0900–CXER0999 | Binding / ABI errors |
**Propagation operators.** Two postfix sigils control how errors flow past an expression:
- `expr?` — **silent propagate.** Call `expr`; if it errored, return the error unchanged through the enclosing directive. Equivalent to Rust's `?`, Swift's `try?`, Haskell's `<-` in the Either monad. The error flows; the caller decides what to do with it. This is the default composition operator.
- `expr!` — **raise at boundary.** Call `expr`; if it errored, raise as a host-native exception immediately (Python `CXError`, Rust `Result::Err` unwrapped to panic, Go `panic(cxErr)`). Used at the binding boundary when you want CX errors to surface to the host runtime's standard error-handling mechanism.
**Handling.** Recovery is `[?match]` (see match) dispatching on the `[err]` channel. The scrutinee is the errorable expression itself; an `[err]` it produces is captured as the match value (not auto-propagated), so a `[case [err …] …]` arm can inspect its code. Arms are tried top-down, first match wins; a bare `$err` capture or `[else …]` is the catch-all. Match by plain attribute equality (`[err code='CXER0204']`) or capture the code (`[err @code=$c]`).
[?match [?let [= $u [$lookup-user $id]?] [$render $u]]
[case [err code='CXER0204'] [err message='no such user']]
[case [err code='CXER0500'] [err message='service unreachable']]
[else [err message='lookup failed']]]
**Pattern-match on errors.** Because an error is a value, you can `[?match]` on its shape rather than its code. Useful when multiple error codes carry the same structured payload (e.g. every `CXER05xx` carries a `:url`) and you want to dispatch on a payload field:
[?let [= $result [err url='https://api.example.com/users/9' status=404]]
[?match $result
[case [err @url=$u status=404] [missing $u]]
[case [err @url=$u status=503] [retry-later $u]]
[case [err @url=$u] [permanent-failure $u]]
[else $result]]]
**Compose with resilience directives.** `[?retry]`, `[?timeout]`, `[?fallback]`, `[?circuit-breaker]` (see resilience) all consume and produce error values. A `[?retry max=3 backoff=exp(100ms)]` wraps an inner expression; if it errors three times the final error flows out as the retry's value. No exception type to catch — the retry directive is a normal expression returning the operand's value-or-error.
Errors-as-values is what makes CX safe to use in pipelines (`|`), in async (`[?await]`), and across thread boundaries (`[?channel]`) — the error is just a value, it doesn't "stack-unwind" through workers. A `[?worker]` that produces an error sends that error through its result channel as a value; the receiver's `[?receive]` sees the value-or-error and the worker's runtime terminates cleanly.
**Why not exceptions.** Exceptions force every function signature to be implicitly contaminated with "may throw," and every caller to either catch or be transparent. Tools cannot see this contamination from the signature. In CX, an error is a value, and an expression that may produce one is an expression of an error-or-value union. The type is in the shape; the propagation discipline is in the operators (`?` and `!`); the handling is in `[?match]`. The cost of an error path is the cost of allocating one value — no stack scan, no frame skipping, no unwind table.
Pure-functional updates and lenses
`[?modify]` (see modify) is CX's lens primitive — name a location, name a change, return a new value. The original is unchanged. The implementation strategy that makes this affordable on large trees is **structural sharing**.
**The technique.** When `[?modify]` produces a new tree, only the spine from the root to each matched node is copied. Every unchanged sub-tree is shared (by pointer) with the original. For a tree of depth `d` with `N` matched nodes, the new heap cost is `O(d × N)`, not `O(size-of-tree)`.
[# Before modify: #]
input doc: output doc (after :set on root/foo/@bar):
root [a] root' [a] <-- new
|- foo [b] |- foo' [b'] <-- new
| |- bar=1 ---+ | |- bar=2 <-- new
| |- baz ---+--shared---| |- baz <-- same ptr
| '- children --+ | '- children <-- same ptr
'- qux '- qux <-- same ptr
The spine `root -> foo` is copied (two element-headers + one attribute map). `baz`, `children`, and `qux` retain their input pointers. The two documents differ structurally — `equals` returns `false`, hashes differ — but share ~99% of memory. This is the same persistent-data-structure pattern as Clojure's PersistentHashMap, Haskell's lens-based zippers, and Rust's `im` crate.
**Why it matters.** Pure-functional updates are normally considered too expensive for large trees. With structural sharing they are competitive with in-place mutation, and they carry every advantage: thread-safe by construction, trivially cacheable, easy to undo, safe to share across module boundaries, deterministic across machines. The conformance gate (30.5 in [`spec/v0_8_0_status.md`](../../spec/v0_8_0_status.md)) measures **< 1 KB new heap per matched node on a 10 MB document**. That is the contract; bindings can rely on it.
**Lens vocabulary.** `[?modify]`'s eleven action slots cover the lens operations from Haskell's `lens` library and Clojure's `assoc-in` / `update-in`:
| action | lens-equivalent |
|---|---|
| [set …] | assoc / set |
| [using …] | update / over |
| [delete] | dissoc |
| [append …] | conj / over (++) |
| [prepend …] | cons / over (:) |
| [insert …] | insertAt |
| [rename …] | renameKey |
| [replace …] | set (whole node) |
| [set-attr …] | assoc-attr |
| [delete-attr …] | dissoc-attr |
| [wrap …] | wrap / under |
[# Real-world example: update every active user's last-seen
timestamp atomically. The original $db is unchanged. #]
[db [user active=true last-seen='2026-01-01T00:00:00Z']
[user active=false last-seen='2025-12-01T00:00:00Z']]
[?let [= $db [$first //db]]
[= $updated [?modify $db
//user[= $_@active true]/@last-seen
[set '2026-07-14T00:00:00Z']]]
$updated]
[# $db and $updated share every inactive user's storage; only
the spine + the modified attribute leaves are new. #]
**Identity invariant.** The input document is observably unchanged after `[?modify]`. No Layer-1 method mutates a Node in place. `Node` and `Doc` are immutable values from the perspective of every binding. The C ABI surface enforces this by not exposing a mutating accessor — there is no `cx_node_set_attribute` in `include/cx.h` and there never will be.
**GC tracking.** Shared subtrees live until every document that references them is unreachable. The V core uses Boehm GC; bindings inherit. Python references via FFI pointer with a `cx_doc_ref` root; Rust uses `Drop` to release the ref; Go uses `runtime.SetFinalizer`. The shared-subtree invariant is binding-agnostic: any two `Doc` values resulting from the same `[?modify]` chain can be freely passed across binding and thread boundaries; libcx tracks the underlying graph.
**Implementation status.** The `[?modify]` surface is currently shipped with full-copy semantics; the spine-copy optimisation and formal perf gate enforcement are planned. Functional semantics are identical between the two — only the allocation profile tightens.
Streaming
Large CX documents — log streams, JSON exports, telemetry dumps, multi-GB sensor traces — can be evaluated incrementally. The `[stream]` clause on `[?for]` tells the evaluator to use a stream-aware parser and apply the body row-by-row instead of materializing the full tree. The streaming spec is [`spec/streaming.md`](../../spec/streaming.md).
[# Stream over the entries — bounded resident memory: the
iteration applies the body row-by-row instead of holding
the whole selection. #]
[logs [entry level='error' service='api']
[entry level='info' service='api']
[entry level='error' service='db']]
[?for [in $row //logs/entry] [stream]
[where [= $row@level 'error']]
[yield $row@service]]
**Event model.** The streaming parser emits a sequence of 14 event types (StartDoc, StartElement, Attr, Text, Scalar, EndElement, EndDoc, plus chunked-table events and pull-callback control events). Every parsed node produces one or more events. Consumer code processes each event and discards it before the next is read. Full event reference in [`spec/streaming.md`](../../spec/streaming.md) §1.
**Threshold.** The streaming path activates automatically for inputs larger than `CX_STREAMING_THRESHOLD` (default **8 MiB**). Below this, the buffered parser is faster (no per-event dispatch overhead). Above, the streaming parser wins on memory and parallel-pipeline throughput. Override the threshold per-call via `cx.parse(stream=True)` or globally via the environment variable.
**Measured throughput.** The Y6 streaming evaluator hits **353 MB/s sustained** on a JSON-shape workload (gate 15 in [`spec/v0_8_0_status.md`](../../spec/v0_8_0_status.md)) — about 17% under the comparable raw JSON parser on the same input. The benchmark is at `vcx/tests/runners/streaming_bench_json.v`; reproduce with `make bench-streaming`. The toy 25-byte `id:name;` corpus (`streaming_bench.v`) hits ~177 MB/s but is loop-control- bound and not representative of real workloads.
**Memory budget.** The parser holds one in-flight event plus a scratch buffer of ~32 KiB. Total resident memory for a streaming pipeline is bounded by:
- **Parser scratch** — ~32 KiB regardless of input size.
- **Active event** — the current StartElement or Scalar, typically < 1 KiB.
- **User accumulators** — whatever the body of `[?for]` builds (a count, a running sum, a sliding window). Bounded by the user; CX does not impose a cap.
- **Output buffer** — 64 KiB by default for flushing emitted CX bytes downstream. Configurable via `CX_STREAM_FLUSH_BUFFER`.
**Composition with chunked tables.** Streaming integrates with the CXCol chunked-table format (`spec/core/data-bin.md` §3.11): a chunked table reader yields one row group at a time, each with per-column min/max/null-count statistics for predicate pushdown. Perfect for `[?for :stream]` pipelines over columnar workloads — a 10 GB Parquet file becomes a constant-memory CX pipeline.
**Streaming write.** The dual surface — emit events without building a tree — is the `cx_events_writer_*` family (ABI cap bit 27, see `spec/streaming.md §6`). 21 C ABI symbols covering 14 stream events × 6 output formats with emit-time validation. Used by builders that want to ship CX to a wire without ever holding the document in memory: a serialiser walks its input shape, calls `cx_events_writer_start_element(...)`, and the bytes flow out. The writer enforces grammar invariants (no scalar after `EndElement`, no attribute after `Text`) at emit time.
**Streaming is not free.** Random access by CXPath is much slower on a stream — you cannot jump to `//user[99]` without walking the previous 99. Streaming is for **forward passes** with bounded look-ahead; if your computation needs random access, parse the whole doc into a tree. The `[stream]` clause is a hint, and the evaluator may materialize a subtree internally if your `:where` predicate requires it.
Parallelism by construction
Because every value is immutable and every error is a value, CX gets parallelism almost for free. Every parallel-capable directive composes via the `[par]` clause — `[?for]` for comprehensions, `[?map]` for higher-order map, `[?reduce]` for associative folds; the async and channel surfaces (see concurrency-model) cover the long-running and message- passing cases. There is no user-level lock primitive — there is no shared mutable state to protect. `[par]` is unordered-by-default for speed; opt into source-order preservation by adding `[ordered]`.
[# Embarrassingly parallel — [par] on [?for] #]
[site [request 'r1'] [request 'r2']]
[?for [in $r //request] [par] [yield [got [$text $r]]]]
[# Higher-order map — add [ordered] to preserve source order #]
[?map (1, 2, 3, 4) [using [?fn ($x) [* $x $x]]] [par] [ordered]]
[# Reduce — associative parallel fold #]
[?reduce (1, 2, 3) [using [?fn ($a $b) [+ $a $b]]] [init 0] [par]]
**Status.** The `[using FN]` clause-child form is the sole accepted shape for `[?map]`/`[?reduce]`/`[?par-map]`; there is no legacy colon-slot form (the current surface is uniformly `[head …]`, never `:slot`).
**Why this is safe.** No CX expression can mutate a value another thread is reading. `[?modify]` returns a new tree; bindings cannot patch a Doc in place. The runtime can split work across threads and re-combine results without synchronization primitives in user code. The C ABI declares every conversion symbol as **(S) stateless** in [`spec/abi.md`](../../spec/abi.md) §1.5.1 — concurrent calls on disjoint inputs are safe without external synchronization; concurrent calls on the same input buffer are also safe because the input is not mutated.
**Determinism.** `[par]` does not guarantee execution order, but it does guarantee the **result** is order-independent — the evaluator only parallelizes over comprehensions whose body is pure (every CX body is pure unless it crosses a service / worker boundary, which is itself explicit). The per-iteration value depends only on the iteration's input; the recombination is associative (concatenation for `[?for]`, user-supplied function for `[?reduce :par]`).
**Worker pool model.** Under the hood, `[par]` dispatches into a bounded worker pool sized to the host's CPU count by default. The pool is per-process, not per-call — multiple concurrent `[par]` invocations share the same pool, capped at `CX_PAR_WORKERS` (default `num_cpus`). For workloads that need a smaller pool (resource-constrained, embedded), set the env var or pass `:max-concurrent` to the directive.
**Cancellation across `[par]`.** If one parallel iteration errors and the surrounding expression short-circuits on that `[err]` (via `?` propagation or a recovering `[?match]`), in-flight iterations are cooperatively cancelled. Each iteration polls a cancellation flag at safepoints (between CXPath steps, between directive invocations, at every `[?check-cancel]`). A long-running CPU-bound iteration without internal safepoints will run to completion — the same hazard as every other cooperative-cancellation system.
**When `[par]` doesn't help.** Tiny iterations (sub-microsecond body) lose to the dispatch overhead. The evaluator's heuristic is to skip parallel dispatch when the input sequence has fewer than 32 items or when the body is detectably trivial (single CXPath step, single attribute access). For pathological cases, `[par]` is planned to accept a batching hint to force the threshold.
Security model and untrusted-input handling
CX is a parser, an evaluator, and an FFI surface — three attack-surface categories, each with its own threat model. The normative reference is [`spec/process/threat-model.md`](../../spec/process/threat-model.md); this section is the developer-facing summary of what an adopter needs to know when CX is in the path of untrusted bytes.
**Deployment shapes.** The threat model defines three:
| shape | source | hardening |
|---|---|---|
| A — Trusted inputs | Operator-authored files / internal data exchange | Default config is sufficient |
| B — Semi-trusted inputs | PRs, CI artifacts, third-party config | Pin libcx version + apply resource limits |
| C — Untrusted inputs | Webhook bodies, public uploads, network payloads | --profile untrusted + process isolation + monitoring |
**Untrusted-input checklist.** If a CX document comes from an attacker-controlled source, the operator MUST:
- **Apply `--profile untrusted`** at the CLI or `cx.config(profile='untrusted')` in the binding. This preset clamps every resource limit to roughly 10% of default (see quotas-recap) and disables include capabilities that read outside the document's directory.
- **Validate against a schema.** A `.cxs` schema (see schema) is the input gate — every field is type- checked, every cardinality is bounded, every regex is RE2-linear. Reject inputs that fail validation before passing them downstream.
- **Isolate the process.** OS-level memory caps (`cgroups`, `ulimit -v`), wall-clock timeouts, and file descriptor caps. CX's internal budgets bound the worst-case per-call cost but do not bound accumulated cost from a malicious sequence of small operations.
- **Disable network includes.** The `include-network` capability bit is off by default; verify it stays off when `--profile untrusted` is in effect.
- **Sanitize error messages.** libcx errors carry line / column numbers and the offending construct's name, never file paths or runtime state. Bindings layer their own context onto the libcx error; consumer code that re-exports CX errors to an external surface should strip caller-supplied path information first.
**Include-resolver capability gating.** The `[?cx include path=...]` directive is the highest-leverage attack surface in CX — a malicious document can request a read from anywhere the host process has filesystem access. The resolver gates includes by capability bit; the default profile enables only `include-local`. See includes for the four bits and the env-var overrides.
| capability | what-it-allows | default |
|---|---|---|
| include-local | Reads under the including document directory tree | enabled |
| include-absolute | Reads anywhere on the host filesystem | disabled |
| include-network | HTTPS URLs as include paths | disabled |
| external-entity | XML external entity references (&foo;) | disabled |
**Entity-expansion attacks (billion laughs).** The classic XML-entity DoS — `<!ENTITY lol "lol"><!ENTITY lol2 "&lol;&lol;…">` recurses to exhaust memory — is mitigated by the `CX_MAX_ENTITY_EXPAND` limit (default 10,000 characters total expansion). Exceeding it raises **CXER0145**. The default is conservative; high-volume legitimate-entity workloads can raise it, untrusted contexts should lower it further or disable entity expansion via the `external-entity` capability bit.
**Resource limits.** Every limit listed in limits is enforced unconditionally; they are not advisory. The full table:
| limit | default | env var | error |
|---|---|---|---|
| Max document depth (nested brackets) | 512 | CX_MAX_DEPTH | CXER0142 |
| Max attributes per element | 1024 | CX_MAX_ATTRS | CXER0143 |
| Max body byte length per element | 256 MiB | CX_MAX_BODY | CXER0144 |
| Max include depth | 16 | CX_INCLUDE_MAX_DEPTH | CXER0141 |
| Max entity expansion | 10000 chars | CX_MAX_ENTITY_EXPAND | CXER0145 |
| Max recursion depth | 1024 | CX_MAX_RECURSE | CXER0146 |
| Max parse time per document | 30 s | CX_MAX_PARSE_TIME | CXER0147 |
| Max canonical-form size | 1 GiB | CX_MAX_CANON | CXER0148 |
| Max function-call recursion | 256 | CX_MAX_CALL_DEPTH | CXER0010 |
| Max materialized sequence length | 1,000,000 | CX_MAX_SEQ_LEN | CXER0011 |
**Hardening already in place (per threat-model §5).**
- **Linear-time regex.** Every regex callsite (schema `pattern=`, `fn:matches`, `fn:tokenize`, `fn:replace`, `fn:analyze-string`) routes through the vendored RE2 shim. Catastrophic-backtracking patterns like `(a+)+$` terminate in linear time. The schema validator's `:pat=` constraint is on the same engine; cross-binding regex flavor drift is eliminated.
- **External-entity rejection by default.** DOCTYPE is parsed but inert. XML XXE and billion-laughs are by-construction immune unless the `external-entity` capability is explicitly enabled.
- **Length-prefixed FFI framing.** Every binary buffer crossing the C ABI is `[u32 LE size][payload]`. Each binding deserializes once with explicit bounds checks before allocation. No unsafe pointer arithmetic outside audited marshalling shims.
- **UTF-8 validation.** Invalid UTF-8 in any input is CXER0001; the parser does not silently replace invalid bytes.
- **Bounded error messages.** libcx errors are length- capped and include no file paths or runtime state.
- **Strict xs: constructors.** `xs:integer(...)` and similar raise CXER0103 / FORG0001 on un-parseable string inputs rather than silently coercing to 0. Closes a category of validation-bypass bugs in caller-written guard code.
- **Reproducible builds + nightly fuzz.** `SOURCE_DATE_EPOCH` + pinned toolchain produces byte-identical libcx artifacts across machines; a 1h/night fuzz workflow exercises the parser + buffered + streaming + ABI surfaces; crashes upload as CI artifacts.
**CXPath comparator-cardinality errors.** When a CXPath general comparator (`=`, `!=`, `<`, etc.) is applied to a sequence whose cardinality doesn't match the comparator's contract, the result is **CXER0103** with the offending operand identified. This matters for untrusted input: a crafted document that pushes a CXPath predicate into an unexpected shape (e.g. an attribute that the schema says is scalar but the input ships as a sequence) is caught at predicate-evaluation time rather than producing a silently wrong result downstream.
**Sandboxing untrusted CX code.** When the *code* itself comes from an untrusted source (not just the data), CX offers a capability model documented in sandboxing. Briefly: the evaluator can refuse `[?http-client]`, filesystem `[?cx include]`, `[?worker]` spawn, and service minting via per-capability flags. Untrusted code by default can compute, transform, hash — it cannot reach the network, the disk, or spawn concurrency. Effect typing as a compile-time mechanism is a future design direction (see sandboxing).
**External audit pending.** CX has not received an external security audit. The hardening above is real and tested through conformance, fuzz harness, and reproducibility CI, but until the audit lands the recommended posture for deployment shape C (untrusted) is process isolation plus the `--profile untrusted` preset. See [`spec/process/threat-model.md`](../../spec/process/threat-model.md) §1 for the full caveat and §6 for the gaps table.
Performance characteristics
CX is fast where it matters and honest where it doesn't. This section is the measured-numbers view: thresholds, throughput, memory profiles, and the comparison table that tells you when CX wins over JSON or YAML and when it doesn't. Normative perf gates are tracked in [`spec/v0_8_0_status.md`](../../spec/v0_8_0_status.md); workloads and benches in [`spec/streaming.md`](../../spec/streaming.md) and [`spec/governance.md`](../../spec/governance.md) §6.
**Throughput by operation.** Order-of-magnitude on a modern laptop (M2 Pro, libcx `-prod`, single-thread):
| operation | 1KB | 1MB | 100MB |
|---|---|---|---|
| cx_to_ast_bin (parse) | < 50 us | < 30 ms | < 3 s |
| cx_to_data_bin (binary) | < 50 us | < 30 ms | < 3 s |
| cx_to_json (project) | < 100 us | < 60 ms | < 6 s |
| cx_events_next per event | < 1 us | — | — |
| cx_select (CXPath) | < 10 us | < 60 ms | < 6 s |
| cx_hash (SHA-256 canon) | < 50 us | < 25 ms | < 2.5 s |
**Streaming evaluator.** Sustained **353 MB/s** on a JSON-shape workload (gate 15 in [`spec/v0_8_0_status.md`](../../spec/v0_8_0_status.md)). About 17% under the comparable raw JSON parser on the same input — which is CX's headline number: type-tagged, schema-validating, structurally-shared, content-addressable data parsed at within-spitting-distance of an untyped JSON dump. The bench is at `vcx/tests/runners/streaming_bench_json.v`.
**Streaming threshold.** `CX_STREAMING_THRESHOLD` default **8 MiB**. Below this, the buffered parser wins. Above, the streaming parser wins. The threshold is empirically tuned; per-workload measurement may justify a different setting, and the env var override is the supported tuning knob.
**Memory budget per Doc.** A parsed `Doc` value carries:
- **Resident allocation** for the AST tree — typically **2-4× the source byte size** (element headers + name interning + attribute maps + body sequences). A 10 MB source document occupies ~30-40 MB resident.
- **Scratch buffer** during parse — ~32 KiB regardless of input size. Released after `Doc` construction.
- **Shared subtree pool** — when multiple `Doc` values are derived from a common ancestor (via `[?modify]`), the shared subtrees are counted once. A pipeline of 10 modifications on a 10 MB doc occupies ~12 MB total, not 100 MB.
- **Name-intern pool** — element and attribute names are interned; a million `[user email='...']` elements share one allocation for `user` and one for `email`.
**Parser scratch.** ~32 KiB regardless of input. Allocated on parse entry, released on parse return. The same scratch buffer is reused across the streaming-parser event loop, so a long-lived streaming parse keeps a constant 32 KiB in-flight regardless of input size.
**Throughput per format.** Roughly, on a parse:
| format | throughput | notes |
|---|---|---|
| CX (text) | ~250 MB/s | Buffered; native parser |
| CX (binary, ast_bin v6) | ~600 MB/s | No tokenizing; direct varint decode |
| CX (binary, data_bin) | ~800 MB/s | Schema-driven; field tags omitted |
| CXCol (chunked table) | ~1.2 GB/s | Columnar; predicate pushdown |
| JSON (cx_from_json) | ~400 MB/s | Less validation, weaker types |
| YAML (cx_from_yaml) | ~80 MB/s | Indentation parsing is slow |
| XML (cx_from_xml) | ~180 MB/s | Entity handling overhead |
| Streaming JSON-shape | ~353 MB/s | gate 15; vs ~440 MB/s native JSON |
**When CX wins over JSON.**
- **Large documents.** CX's binary forms (ast_bin v6, data_bin, CXCol) are 30-60% smaller than equivalent JSON on the wire, and parse 2-4x faster. For 100 MB+ telemetry, log, or analytics workloads, CX is meaningfully cheaper to move and process.
- **Repeated parsing.** CX hashes are stable across equivalent canonical forms, so a content-addressed cache hits on byte-identical canonical bytes regardless of whitespace, attribute order, or comment presence in the source. JSON has no equivalent — every formatter produces a different SHA-256.
- **Content-addressing.** Tied to the above. CX is the natural fit for build pipelines, IPFS-style storage, and "did this artifact change semantically?" queries.
- **Type-tagged data.** JSON has no native dates, datetimes, bytes, or sized integers. CX has all eight scalar kinds plus sized variants. Round-tripping a datetime through JSON requires both ends to agree on a string format; through CX it's a first-class scalar.
- **Schema-validated workloads.** A CX schema is itself a CX document; you author it in the same syntax, the validator uses RE2-linear regex, defaults materialize at validation time. JSON Schema is an external ecosystem with its own tooling.
- **Code-as-data.** Shipping a CX program across the wire is shipping a CX document — same projection table, same canonical form, same hash. JSON requires a separate code transport (JS, Lambda, etc.).
**When JSON wins over CX.**
- **Interop with non-CX consumers.** JSON is the lingua franca. If the downstream consumer doesn't know CX, you project to JSON and pay the price (lossy types, verbose, no content-addressing). The cost is the price of compatibility.
- **Tiny one-off payloads.** A 50-byte config blob is the same cost in JSON or CX; JSON has tool ubiquity. The CX win shows up at scale.
- **Pure key-value config files.** TOML or JSON has the edge for `host=...`, `port=...` flat config; CX shines when the shape gets nested or typed.
- **Browser-native consumption.** Until libcx.wasm lands broadly (see wasm-story), JSON is what the browser speaks natively. libcx.wasm closes this gap for in-tree tooling; full browser ecosystem reach is a longer arc.
**Reproducible builds.** libcx ships with reproducible-build CI. Setting `SOURCE_DATE_EPOCH` and pinning the toolchain produces byte-identical artifacts across machines. The same applies to canonical-form output of any CX document — same input → same canonical bytes, regardless of which machine, which OS, which CPU architecture (see determinism).
ABI stability and versioning
CX's C ABI is the contract that lets nine bindings (Python, Go, Rust, V, TypeScript, Java, Kotlin, Swift, C#) call into libcx without recompilation. The ABI is versioned, capability- gated, and append-only at the symbol level. The normative policy is [`spec/governance.md`](../../spec/governance.md) §5; the capability bit map is [`spec/abi.md`](../../spec/abi.md) §3.
**Semver discipline.** CX follows semver:
- **0.x.y** — pre-1.0. The ABI can break at minor versions; the project annotates each break in `CHANGELOG.md` and provides a migration guide. v0.8.0 [; version-literal-ok ] renamed the evaluator symbol prefix (`cx_program_*` → `cx_code_*`); callers update via the migration script in from-v07x.
- **1.0** — the API/format-stability boundary through the 1.x line. Once 1.0 ships, ABI breaks happen only at major version bumps and with a published migration guide.
- **N.x.y** for N ≥ 1 — minor versions add features (advertised by new cap bits); patch versions fix bugs. Neither breaks ABI.
**Capability bits as forward-compat.** The 64-bit `cx_features` bitmask is the runtime-detection mechanism. A binding examines the bitmask on load and refuses to use features the loaded library doesn't implement. New features get a new bit; the old bits keep their meanings forever. The bit map (as currently defined):
| bit | hex | capability |
|---|---|---|
| 0 | 0x00000001 | ABI v1 conversion symbols |
| 1 | 0x00000002 | Binary AST (cx_to_ast_bin) |
| 8 | 0x00000100 | CXPath C ABI |
| 9 | 0x00000200 | Real streaming (cx_events_open/next/close) |
| 21 | 0x00200000 | CXCol chunked-table format |
| 23 | 0x00800000 | Apache Arrow C-Data interop |
| 24 | 0x01000000 | Schema-driven encoding |
| 25 | 0x02000000 | Schema validator (RE2-linear regex) |
| 26 | 0x04000000 | Thread-init handshake |
| 27 | 0x08000000 | Streaming-write API |
| 28 | 0x10000000 | CX code evaluator (cx_code_eval) |
| 29 | 0x20000000 | Collection literals + CXDM v1.1 |
| 30 | 0x40000000 | Parameterized templates + ?fn |
| 31 | 0x80000000 | CX program diagram renderer (cx_code_diagram) |
| 32 | 0x100000000 | CX data-tree viz (browser, AST-JSON contract) |
| 33-63 | reserved | Allocated as features land |
**Bit allocation policy.** A new capability bit is allocated for every accepted feature that adds a public ABI surface. The bit is reserved when the spec change is accepted; it goes live (set in `cx_features`) when the implementation lands. Partial implementations leave the bit clear. Removing a bit requires a major libcx version bump. See [`spec/governance.md`](../../spec/governance.md) §5 for the normative policy.
**ABI version handshake.** Every binding calls `cx_abi_version` at load time and compares against the major version it was built for:
- **Equal major version** — proceed normally.
- **Loaded library higher** — log a notice and proceed (later libcx is backward-compatible with earlier binding code at the same major version).
- **Loaded library lower** — fail to load with a clear error message. The binding requires a feature the library doesn't ship.
The handshake is cheap — a single C ABI call returning an integer — and it runs once per binding load. Bindings cache the result; subsequent calls into libcx do not re-check.
**Forward-compatibility: newer libcx + older binding.** Works. The library exports a superset of what the older binding knows; unused symbols and unrequested cap bits sit unused. The binding sees only the surface it was built for. This is the common case after a libcx upgrade — bindings keep working without recompile.
**Backward-compatibility: older libcx + newer binding.** The binding refuses to load if a required cap bit is missing. If the missing cap bit is optional (the binding degrades gracefully without it), the binding loads but disables the corresponding API surface and surfaces a clear "feature unavailable" error if user code calls it. Each binding documents its required-vs-optional cap set in its README.
**Symbol stability.** The set of exported symbols is part of the ABI contract. CI lints exports against a whitelist (`tooling/abi/v2_symbols.txt`). New symbols are added to the whitelist in the same PR. Removed symbols cause CI failure unless the major version is bumped. ABI v1 signatures are frozen forever; ABI v2 signatures are frozen as of v0.8.0 — future signature changes [; version-literal-ok ] introduce `_v3` suffixed symbols without removing v2.
**Schema (.cxs) compatibility.** The schema language itself versions independently. Within a major version of the schema spec, you can safely:
- **Add a new field with a default.** Older readers ignore unknown attributes (in open / strict modes); the default materializes for them at validation time.
- **Add a new optional attribute.** Older writers don't produce it; older readers don't require it.
- **Add a new element type.** Existing documents don't reference it; new documents that do are valid against the new schema.
- **Widen a constraint.** Loosening `max-length` or widening an `enum` is forward-safe; existing-valid documents stay valid.
What is **NOT** safe within a major schema version:
- Renaming a field — breaks every reader.
- Removing a field — breaks downstream consumers that depend on it.
- Tightening a constraint — narrowing `enum`, lowering `max-length`, switching `required=false` to `required=true` — turns existing-valid documents into errors.
- Changing a type — `email :string` to `email :int` is a hard break; the value model is different.
These rules are enforced by `cx schema-diff` — a tool that compares two `.cxs` files and flags every change with a compatibility class (safe / unsafe / new-major). Use it in PR reviews to catch unintended breaks.
**The version-negotiation handshake at FFI init.** Detailed flow:
- **Binding load.** Host runtime (Python interpreter, Go binary, Rust process) loads libcx via the host's FFI mechanism (`ctypes.CDLL`, `cgo`, `extern`).
- **`cx_init` call.** Required for bindings whose host spawns OS threads outside V's control (Rust, C#, Java). Advertised by cap bit 26. Cheap and harmless on every other binding.
- **`cx_abi_version` call.** Returns a packed integer (major, minor, patch). The binding compares against its build-time constant; mismatch → refuse to load with a clear error.
- **`cx_features` call.** Returns the 64-bit cap bitmask as a hex string. The binding parses it, compares against its required-features set, and either enables the corresponding API surface, disables it with graceful degradation, or refuses to load.
- **Per-thread registration.** Threads that did not run `cx_init` call `cx_thread_register` on first call into libcx; cleanup on thread exit calls `cx_thread_unregister`. The Rust binding does this automatically at every FFI chokepoint.
Concurrency model
CX's concurrency model is built on three foundations: every value is immutable (so cross-thread sharing is zero-cost), every error is a value (so propagation crosses thread boundaries cleanly), and structural sharing means derived values share memory with their origins. The user-level surface — services, workers, async, channels — is documented in [`spec/code.md`](../../spec/code.md) §11.
**Pure-functional values are concurrency-safe by construction.** No CX expression mutates a value another thread is reading. `[?modify]` returns a new tree; bindings cannot patch a Doc in place; the C ABI exposes no mutating accessor. A `Doc` value passed to a `[?worker]` over a `[?channel]` is the same `Doc` value the sender holds — both see the same immutable bytes, neither can change them.
**Structural sharing across thread boundaries.** A `Doc` derived from another shares subtrees with the original via pointer equality. Cross-thread sharing has zero per-Node overhead — sending a 10 MB Doc to a worker thread is sending a pointer to the document header; the worker walks the shared subtree without copying. Boehm GC tracks the reachability; when no thread holds the subtree anymore, it's collected.
**Service / worker / async surfaces.** Three compositional layers from [`spec/code.md`](../../spec/code.md):
| layer | directives | what-it-models |
|---|---|---|
| Async | [?async] [?await] [?await-all] [?await-any] [?await-race] [?cancel] [?check-cancel] [?sleep] | Lightweight futures + cancellation |
| Workers | [?worker] [?worker-handle] [?channel] [?send] [?try-send] [?receive] [?try-receive] [?close] [?select] [?stop] | CSP-style workers + bounded channels |
| Services | [?service] [?service-handle] [?http-client] [?stop] | HTTP services + clients with resilience |
[# Spawn three workers, each fetching one URL; collect
results via await-all; cancel any in-flight on first
error. #]
[?match
[?await-all
[?async [$fetch 'https://a.example/data']]
[?async [$fetch 'https://b.example/data']]
[?async [$fetch 'https://c.example/data']]]
[case [err $e] [err message='one or more fetches failed']]
[case $v $v]]
**Lock-free reads.** Because no Node is ever mutated, reads from a Doc are lock-free across any number of concurrent readers. The C ABI declares conversion symbols as **(S) stateless** (`spec/abi.md` §1.5.1); concurrent calls on the same input are safe without external synchronization. This is the foundation of CX's parallelism story — `[?par]` comprehensions over a shared Doc spawn workers that read the same underlying bytes without contention.
**Cancellation propagation.** A `[?cancel]` request on an async handle propagates cooperatively through every downstream `[?async]`, `[?worker]`, and `[?channel]` operation in the cancellation scope. Each iteration polls a cancellation flag at safepoints:
- **Between CXPath steps** — the path evaluator checks before each axis traversal.
- **Between directive invocations** — every call into a sub-directive checks first.
- **At explicit `[?check-cancel]`** — user-inserted poll points inside CPU-bound loops where the evaluator has no implicit safepoint.
- **Inside `[?sleep]`** — sleeps return early on cancellation with a CXER0701 (cancelled) error value.
- **Inside channel ops** — `[?send]` / `[?receive]` wake on cancellation, returning CXER0701.
**Cooperative-cancellation hazard.** A long-running CPU- bound iteration without internal safepoints will run to completion. The same hazard as every other cooperative model (Go contexts, Rust async, Python asyncio). Mitigate by inserting `[?check-cancel]` calls in hot loops, by setting per-call `[?timeout]` budgets, or by structuring workloads so that natural iteration boundaries (CXPath steps, `[?for]` iterations) serve as implicit safepoints.
**Error propagation across thread boundaries.** A worker that produces an error sends that error value through its result channel; the receiver's `[?receive]` sees the value-or-error and the worker's runtime terminates cleanly. No exception escapes the worker boundary; no stack unwinds across threads. The error is a value; values travel through channels by definition.
**Process-level isolation.** Workers within a process share the libcx address space and the Boehm GC heap. Workloads that need stronger isolation (untrusted code, resource-quota enforcement, crash isolation) run in separate OS processes communicating via the CX wire format (data_bin over a socket or pipe). The wire format is self-describing and version-stable; the `cx_events_writer_*` family supports zero-copy emit into a shared-memory ring buffer if the workload justifies the effort.
Wasm / browser story
CX compiles to WebAssembly via libcx.wasm. The wasm build is a strict subset of the native C ABI — every symbol exported is a symbol that already exists in libcx.so / libcx.dylib — so bindings and tools that target the native ABI work identically against the wasm build for the subset of features it exposes.
**Why wasm.** Three motivations:
- **Browser-side tooling.** Documentation playgrounds, schema validators, linters, formatters — running inside the browser without a round trip to a server. The CX playground ( [demo.html](../../tooling/web/demo.html)) is the flagship example.
- **Low-trust embedding.** A wasm sandbox is a hard isolation boundary — the embedded libcx can't reach the host filesystem, network, or other tabs. Useful when the embedding context (a content editor, a third-party widget) needs CX evaluation without granting full process access.
- **Demo and learning surfaces.** Try-before-you-install for tutorials, examples, and onboarding documents. The playground lands the language in a browser tab with no install step.
**Capability surface in wasm.** A strict subset of native. The wasm build does **not** export:
- `[?cx include]` with file-system paths — no filesystem access in wasm.
- `[?http-client]` — no network access (the browser's fetch API is the embedding's responsibility).
- `[?worker]` spawning OS threads — wasm runs in a single browser thread by default.
- The thread-init handshake — cap bit 26 is left clear; wasm callers do not call `cx_init` / `cx_thread_register`.
The wasm build **does** export:
- The full CXDM value model, parser, canonical form, hash, projection table — every cap bit from §9.8 that is purely computational. Bits 0-11, 13-25 are all set.
- **Bit 28** — CX code evaluator (`cx_code_eval`, `cx_code_eval_with_len`, `cx_code_eval_streaming`).
- **Bit 29** — collection literals + CXDM v1.1 container Items.
- **Bit 30** — parameterized templates + `?fn`.
- **Bit 31** — CX program diagram renderer (`cx_code_diagram`) via the wasm-safe Mermaid text emit path. SVG / PNG render formats remain CLI-only (graphviz shell-out) and are not advertised by this bit.
- **Bit 32** — CX data-tree viz contract (browser containment-only tree built JS-side over the `cx_to_json` AST projection).
**cxlib.js — the Layer-1 surface.** A small (~140 LOC) hand-written JavaScript wrapper exposes the wasm C ABI as a clean JS API. Sixteen methods:
| method | what-it-does |
|---|---|
| loads(text) | Parse CX text → Doc handle |
| dumps(doc) | Doc → CX text (formatted) |
| toJson(doc) | Doc → JSON string |
| fromJson(json) | JSON → Doc handle |
| toCanon(doc) | Doc → canonical CX bytes |
| hash(doc) | SHA-256 of canonical bytes (hex) |
| equals(d1, d2) | Value-level equality (hash compare) |
| evalCode(doc, code) | Run a CX code program against doc; return result Doc |
| evalCodeStreaming(doc, code, onChunk) | Streaming eval; chunks delivered to JS callback |
| diagram(doc) | Render a CX program as Mermaid sequence diagram (text) |
| select(doc, path) | CXPath select → result Doc |
| validate(doc, schema) | Validate doc against .cxs schema; return errors [] |
| version() | libcx version triple (major, minor, patch) |
| features() | cap-bit hex string |
| free(doc) | Release a Doc handle (explicit, since JS lacks finalisers in some runtimes) |
| module() | Raw wasm Module for embedding-specific extensions |
**The `<cx-diagram>` web component.** A custom element that embeds a CX program rendered as a sequence diagram. The component takes either an inline `cx` attribute or a `src=...` URL pointing at a .cx file; it loads libcx.wasm, calls `diagram()`, and renders the Mermaid text via mermaid.js (CDN-loaded by default, override-able via `cxDiagramConfig.mermaidUrl` for offline / corporate contexts).
[?for [in $order //order] [par]
[yield [?service-handle name='billing'
[process $order]]]]
**Size budget.** libcx.wasm + cxlib.js together come in at **~1.93 MB** (SINGLE_FILE=1 build). Inflated to 2 MB once the program diagram and tree viz wasm paths are linked. Cacheable in the browser (long-lived Cache-Control header), gzip-compresses to ~600 KB on the wire. The target ceiling is **2 MB** uncompressed; growth past that triggers a linker-level audit.
**When wasm makes sense.**
- **Browser-side tooling.** Playgrounds, validators, diagram renderers, in-page documentation evaluators.
- **Demo sandboxes.** Tutorials, blog posts, "try this snippet" surfaces.
- **Low-trust embedding.** Content editors, third- party widgets, plugin sandboxes where the host wants CX power but not CX's full process access.
- **Server-side wasm runtimes.** Wasmtime, wasmer, etc. — a self-contained CX evaluator that ships as a single 2 MB module.
**When native libcx wins over wasm.**
- **Performance-critical workloads.** Wasm runs at 50-80% of native speed depending on the workload and the wasm runtime. For multi-GB pipelines, native is the right call.
- **File-system or network access.** Wasm in a browser has neither; wasm in WASI has a constrained version of both. Native gives you the host OS's full surface.
- **Multi-threading.** Wasm threading is still evolving (the threads proposal, shared memory). For `[?par]` over a Doc, native gives you full Boehm GC + V threads.
- **Existing native binding code.** If you're already in Python / Go / Rust, native libcx is the obvious choice; wasm is for the contexts where native isn't available.
**Implementation status.** libcx.wasm builds via `scripts/wasm/build_libcx_wasm.sh` and lands in `dist/wasm/`. The playground at [demo.html](../../tooling/web/demo.html) uses cxlib.js to evaluate CX programs live in-browser. Gate-17 design points D1-D11 are shipped (see [`spec/audits/playground_gate17_design_v1.md`](../../spec/audits/playground_gate17_design_v1.md)). Tier-3 (wasm-graphviz for SVG diagrams in-browser) is planned.
**The playground is a constrained demonstration surface, not the performance target.** Bundling CX into a browser carries unavoidable overhead — Asyncify instrumentation (so wall-clock `[?sleep]` can yield through the JS event loop without freezing the UI), pthreads runtime (so `[par]` runs real OS threads), single-file base64 inlining or separate-file fetching, and the wasm format itself. Production CX deployments use the native CLI (~5MB), the language bindings (V / Python / Go / Rust / others — ~1.5MB per wrapper), or libcx as a shared library directly. Native and binding paths have **no Asyncify overhead, no SharedArrayBuffer requirement, no bundle size pressure** — they're the production target. The playground exists to make CX explorable from any browser without an install step; treat its performance characteristics accordingly.
**Three playground delivery modes:**
- **Static via GitHub Pages (HTTPS).** Single-threaded ASYNCIFY build. `[par]` examples produce correct output but run sequentially. No cross-origin isolation possible on Pages (no custom headers); same constraint as file://.
- **Static via file:// (double-click `playground.html`).** Same wasm artifact as Pages; same single-threaded execution. Browser blocks Workers and SharedArrayBuffer under `file://` origin, so no path to real concurrency here.
- **Local HTTP via `make guide-http`.** Boots a V-based `veb` static-file server that sends `Cross-Origin-Opener-Policy: same-origin` + `Cross-Origin-Embedder-Policy: require-corp` on every response. SharedArrayBuffer becomes available → pthreads-enabled libcx-pthreads.wasm loads → `[par]` uses real OS threads. The playground's adaptive banner tells you which mode you're in.
**Future possibility — native JS reimplementation of CX**: a pure-JS CX interpreter (no wasm, no Asyncify, smaller bundle for embedded use cases like editor extensions) is a long-term option, **not currently scheduled**. The wasm path is the strategic primary because it keeps a single source of truth — `libcx`. A separate JS reimplementation would require maintaining two implementations in parity with the spec, a real cost we haven't decided to take on.
Resource quotas (recap)
For operational use against adversarial workloads, set the resource quotas explicitly. The full table lives in limits; this section is the "what to set" recipe.
**Default profile.** Adequate for trusted inputs (deployment shape A). Limits:
- `CX_MAX_DEPTH=512` — generous; most real documents are < 50 deep.
- `CX_MAX_ATTRS=1024` — generous; most elements have < 20 attributes.
- `CX_MAX_BODY=256 MiB` — accommodates large embedded payloads (images, blobs).
- `CX_INCLUDE_MAX_DEPTH=16` — bounded include chain.
- `CX_MAX_ENTITY_EXPAND=10000` — billion-laughs mitigation.
- `CX_MAX_PARSE_TIME=30s` — wall-clock cap per document.
**Untrusted profile (`--profile untrusted`).** Pre-set clamps every limit to ~10% of default. For network-facing services accepting CX payloads:
- `CX_MAX_DEPTH=64` — anything deeper is likely malicious.
- `CX_MAX_ATTRS=128` — anything wider is likely malicious.
- `CX_MAX_BODY=2 MiB` — reject mega-payloads at the parser boundary.
- `CX_INCLUDE_MAX_DEPTH=4` — include chains beyond four levels are usually an attempt to amplify.
- `CX_MAX_ENTITY_EXPAND=1000` — tighter billion-laughs cap.
- `CX_MAX_PARSE_TIME=3s` — anything slower is either malicious or a misuse.
- **`include-network` capability OFF** — no network fetches.
- **`include-absolute` capability OFF** — no filesystem reads outside the document tree.
- **`external-entity` capability OFF** — no XML entity expansion.
**Custom profiles.** For workloads where the defaults are too tight (high-volume internal log processing) or too loose (extreme adversarial input), set the env vars individually before parsing. The bindings expose the same knobs as keyword arguments on `cx.config(...)`.
**Monitoring.** Each enforced limit raises a structured CXER error (CXER0010-CXER0148 range) with the offending value carried in the error payload. Production deployments should ship these errors to monitoring (with the input identifier scrubbed); spikes in CXER0142 (depth exceeded) or CXER0147 (parse-time exceeded) indicate an attack or a workload misconfiguration.
Determinism and reproducibility guarantees
CX guarantees that the same input produces byte-identical output, across machines, OSes, architectures, libcx versions within a major. This is not just a niceness — it's the basis of content-addressing, reproducible builds, cross-binding parity testing, and audit-trail verification. The normative reference is [`spec/canonical.md`](../../spec/canonical.md); cross-binding determinism is gated by [`spec/governance.md`](../../spec/governance.md) §2.3.
**Canonical form.** Every CX value has a unique strict- canonical byte sequence. `cx canonical` emits it; `cx hash` hashes it (SHA-256); `cx eq` compares two values by hashing both and comparing the hashes. The canonical-form rules are normative and binding-independent:
- Attribute order is lexicographic by attribute name.
- Whitespace is collapsed per documented rules (single-space body, no leading or trailing whitespace, no comment preservation in strict form).
- String quoting is the minimum legal form (single- quoted for single-line, triple-double-quoted for multi-line).
- Numeric formatting is the shortest decimal that round-trips (see below).
- Anchors and merges are expanded; the canonical form is the post-expansion shape.
- Namespace prefixes are rewritten to a deterministic sequence (`ns0`, `ns1`, …).
- The encoding is UTF-8 without BOM, NFC-normalized for identifier names.
**Same input → byte-identical output.** Two `cx canonical` invocations on the same CX value produce the same bytes:
- **Across runs.** Re-running `cx canonical doc.cx` on the same machine, with the same libcx version, gives byte- identical output. Tested by every conformance run.
- **Across machines.** Same canonical bytes on different machines (assuming same libcx version). Tested in cross-binding determinism CI (`spec/governance.md` §2.3).
- **Across architectures (x86-64 vs ARM64, little vs big-endian).** Same bytes. Float formatting uses CX's shortest-form algorithm (independent of `printf`); int encoding is value-based, not memory-layout-based.
- **Across operating systems (macOS, Linux, Windows, BSD).** Same bytes. No path-separator dependence in canonical form; no locale dependence in number formatting; no line-ending dependence.
- **Across libcx versions within a major.** Same bytes for inputs valid against the older version. Bumping libcx within the 0.x line does not change the canonical form of any document valid against the prior version.
- **Across bindings.** Python `cx.canon(doc).hash()` equals Go `cx.Canon(doc).Hash()` equals Rust `doc.canon().hash()`. Gate 28.6 (Layer-1 byte- identical parity) is the conformance test.
**Float formatting.** CX implements its own shortest-round-trip algorithm (Grisu / Ryu family) for canonical float emission, not the host's `printf`. This is load-bearing for cross-machine determinism — `printf("%g", x)` produces different output on different libc versions (musl vs glibc vs Apple libsystem). The CX algorithm is self-contained, version-stable, and documented in `spec/canonical.md` §4.2.
**IEEE-754 strict arithmetic in [?modify [using …]].** CX arithmetic inside `[using …]` lambdas is IEEE-754 strict: same rounding (round-to-nearest-even), same NaN handling, same denormal handling, same exception flags. The evaluator does not enable fast-math; results are bit-identical across platforms. Sin / cos / log / exp use libm; libm determinism is platform-dependent, so the canonical-form guarantee covers exact-arithmetic operations (`+`, `-`, `*`, `/` on `:int` and `:float`) and the shortest-form output formatter. Transcendentals are documented as platform-dependent.
**Hash stability.** `cx hash` is SHA-256 of canonical bytes. Stability follows from canonical-form stability: same input → same canonical bytes → same hash. Two semantically distinct CX documents producing the same SHA-256 is an attack on SHA-256, not on CX. The CX format contributes no second-preimage weakness beyond the underlying hash. Gate 28.6 in [`spec/v0_8_0_status.md`](../../spec/v0_8_0_status.md) enforces this across every binding.
**Reproducible builds.** libcx itself is built reproducibly. Setting `SOURCE_DATE_EPOCH` (e.g. to the source commit's timestamp) and using the pinned toolchain (V version + emcc version + system compiler ABI) produces byte-identical libcx artifacts across machines. The CI workflow (`.github/workflows/reproducibility.yml`) double-builds from the same source on different runners and asserts SHA-256 equality. See [`docs/reproducible_builds.md`](../../docs/reproducible_builds.md) for the full procedure.
**Why determinism matters operationally.**
- **Build caches.** Content-addressed build artifacts hit the cache on byte-equality, regardless of whitespace, comment changes, or attribute order in the source.
- **Diff workflows.** Two semantically-equivalent CX documents produce identical canonical forms — code review sees only semantic diffs, not formatting noise.
- **Audit trails.** A CX document signed at time T verifies at time T+N because the canonical form is version-stable. The signed bytes are the canonical bytes; equality is preserved.
- **Cross-binding tests.** A test fixture authored against the V reference verifies identically against Python, Go, Rust. Gate-28 conformance fails loudly when a binding drifts.
- **Reproducible deployments.** Build the same libcx binary from source on the deployment host; verify the SHA-256 matches the distribution `dist/SHA256SUMS.txt`; ship with confidence.
**What is NOT deterministic.** A few well-flagged carve-outs:
- **`[par]` execution order.** Per parallelism, parallel iterations may complete in any order. The *result* is order-independent (the recombination is associative); the per-iteration timings are not.
- **`now()` / `random()` / `uuid()`.** Effectful builtins with externally-sourced state. Marked as `SideEffect` in the function catalog; refused under `[?cx pure-only]`.
- **Wall-clock timeouts.** Per-call `[?timeout]` budgets are clock-dependent; a slow runner can produce different behavior than a fast one. Mitigate by setting timeouts generously in tests.
- **Transcendental floats.** `sin`, `cos`, `log`, `exp` etc. use libm; libm is platform-dependent. For full cross-platform determinism, use a CX extension that vends its own implementations.
Sandboxing untrusted CX code
When the **code** being evaluated comes from an untrusted source (not just the data), CX offers a capability model that bounds what the code can reach. The default profile is permissive (operator-authored programs are trusted); the `--profile untrusted` preset is the locked-down mode. Normative reference: [`spec/process/threat-model.md`](../../spec/process/threat-model.md) §10 (cx_code_eval trust model) and §11 (cx:eval trust model).
**Two threat surfaces.** CX distinguishes between:
- **Untrusted data, trusted code.** The CX program is operator-authored; the data document is from a network. The hardening from security applies — resource limits, RE2-linear regex, schema validation at the boundary. The threat is DoS or malformed-input; the code itself is safe.
- **Untrusted code.** The program is from a network (a CMS lets editors author templates; a third-party widget ships its own logic; a webhook payload contains a CX expression to evaluate against caller data). The threat is **what the code can reach** — filesystem, network, other processes.
**The capability model.** Untrusted code is gated by a set of opt-in capabilities. The default `--profile untrusted` starts with **no capabilities**; the operator grants specific ones via `--cap` flags:
| capability | flag | what-it-grants |
|---|---|---|
| Filesystem-local include | --cap include-local | `[?cx include path=...]` for paths under the document directory tree |
| Filesystem-absolute include | --cap include-absolute | `[?cx include path=...]` for paths anywhere on the host |
| Network include | --cap include-network | `[?cx include path=https://...]` for HTTPS URLs |
| External entity | --cap external-entity | XML external entity references |
| HTTP client | --cap http-client | `[?http-client]` directive can be constructed and used |
| Worker spawn | --cap worker | `[?worker]` can spawn concurrent OS threads |
| Service mint | --cap service | `[?service]` can bind a network port |
| Mutable channels | --cap channel | `[?channel]` / `[?send]` / `[?receive]` for cross-thread state |
**What untrusted code CAN do (no capabilities granted).** The default `--profile untrusted` permits pure computation. Specifically:
- **Compute over inputs.** Every CX expression — CXPath selection, FLWOR, arithmetic, string manipulation, date arithmetic, regex matching (RE2-linear) — is available. The evaluator cap on call depth (CXER0010), sequence length (CXER0011), and parse time (CXER0147) bounds the worst-case cost.
- **Transform data.** `[?modify]` is pure-functional; it can rewrite any CX value into another. No external effect.
- **Hash and canonicalize.** `cx-hash`, `cx-canon`, `cx-eq` are pure. Untrusted code can compute hashes over inputs and compare them.
- **Pattern-match and dispatch.** `[?match]`, `[?if]`, `[?else]` — all pure.
- **Project to JSON / YAML / TOML / XML / Markdown.** All projections are pure transformations; no I/O.
- **Read its arguments.** The caller-supplied context (`cx:eval`'s context map, the data document passed to `cx_code_eval`) is fully visible. The threat model names this explicitly — the caller chooses what to put in the context.
**What untrusted code CANNOT do (without explicit grants).**
- **Read files.** `[?cx include]` returns a documented "not allowed" error (CXER0820 — capability denied). The regression `test_u1_include_path_traversal_blocked` asserts no syscall happens before the refusal.
- **Reach the network.** `[?http-client]` cannot be constructed; the directive returns CXER0820 at evaluation time. No `getaddrinfo`, no `connect`, no `TLS`.
- **Spawn concurrency.** `[?worker]`, `[?async]`, and `[?service]` directives all refuse without their capabilities. The evaluator continues single-threaded.
- **Mint services.** `[?service]` requires `--cap service` to bind a port; without it, refused at parse time.
- **Read environment variables / process state.** No `[?cx env]` directive exists; environment is not in scope of CX programs by design.
- **Escape resource budgets.** Recursion depth, sequence length, parse time — all bound by limits in quotas-recap. Untrusted code cannot raise its own limits.
**Per-capability grants.** Operators grant capabilities selectively. A common pattern is "let the code read its template includes, but nothing else":
# Grant only local includes, nothing else:
cx untrusted.cx --profile untrusted --cap include-local
# Grant local includes + HTTP client, but not absolute
# filesystem or worker spawn:
cx untrusted.cx --profile untrusted \\
--cap include-local \\
--cap http-client
**The five mitigations for `cx:eval`.** When a CX program calls `cx:eval(source, context)` — i.e. the program itself evaluates a runtime-supplied source string — five mitigations apply:
| mitigation | error | defends-against |
|---|---|---|
| M1: requires [?cx allow-eval=true] | CXER0041 | Accidental exposure — silent weaponization |
| M2: incompatible with [?cx pure-only] | CXER0042 | Defense-in-depth — pure-only blocks side effects |
| M3: sandboxed [?eval] environment — context-map only | no code, visibility check | Information disclosure / capability escalation |
| M4: module pass-through is narrowing | CXER0043 | Module enumeration / privilege escalation |
| M5: recursion-depth cap (default 8) | CXER0044 | Stack-exhaustion DoS via recursive cx:eval |
**Recommended deployment patterns for untrusted-eval workloads.**
- **Process boundary.** Run untrusted-code workloads in a separate OS process with cgroups / `ulimit -v` / equivalent. The capability model bounds worst-case- per-call but does not bound accumulated wall-clock from many cheap operations.
- **`pure-only` at the document head where possible.** `[?cx pure-only]` blocks every SideEffect-tagged builtin. Combined with the process boundary, it gives defense-in-depth: an attacker who somehow widens the capability set still cannot reach side effects.
- **Lint `[?cx allow-eval=true]` presence.** `cx lint` emits informational `L006-eval-bearing` on every document carrying the flag. Elevate to error in deployment via `.cxlint.cx`.
- **Context-map minimisation.** Pass only what the fragment needs. M3 means the context map literally enumerates the fragment's reach into caller data.
- **Module-set minimisation.** Activate only required modules. M4 means the fragment cannot widen beyond the caller's set.
**Effect typing — future direction.** The capability model above is **runtime**: capabilities are checked when the offending directive is evaluated, not at parse time. The complementary design — **compile-time effect typing** — is a future direction. Each directive would carry a static effect signature (`Pure`, `IO`, `Net`, `Spawn`, `Mutate`); the type checker would reject a `--profile untrusted` program that statically references an ungranted effect, regardless of whether that reference would be reached at runtime.
The advantages of compile-time effect typing: errors at load time rather than mid-execution; tools can prove a program effect-safe without running it; auditable effect-set per program. The disadvantages: every directive needs an effect annotation; user-defined functions need effect inference; macro-style runtime construction of directives becomes harder to type. Status: design level per [`spec/process/threat-model.md`](../../spec/process/threat-model.md) §11.4 (no CPU/memory budget per fragment — the runtime check is what's load-bearing today). The current surface ships the runtime capability model; effect typing waits.
**Known limits.**
- **No CPU/memory budget per `cx:eval` fragment.** The mitigations bound recursion depth and module enumeration; they do not bound total cycles or allocations spent inside one `cx:eval` call. A fragment that runs `fn:fold-left` over 1M items is not refused by M1-M5 alone. Process-level limits are the load-bearing defense.
- **Outputs are not auto-sanitized.** `cx:eval` returns a CX value; `cx:render` returns a string. HTML contexts MUST route through `[?cx output-target=html]` (auto-escape) or apply a domain-appropriate sanitizer.
- **Origin threading is informational.** The `options.origin-uri` / `origin-line` keys thread through to error reporting but are caller-asserted; a fragment cannot rely on them for security decisions.
- **External audit pending.** CX has not yet received an external security audit. The capability model is real and tested through conformance; the audit closes residual confidence gaps. See security for the timeline.
**Reference.** The full normative threat model — including the M1-M5 mitigations and their error codes — is [`spec/process/threat-model.md`](../../spec/process/threat-model.md). The per-capability flag set is exposed through CXER0820 (capability denied) and the `cx` CLI's `--profile untrusted` preset.
Structural sharing — the spine-copy algorithm
Why `[?modify]` is fast even though every update returns a new document. The spine-copy + share-everything-else algorithm.
The puzzle
CX documents are immutable values. Every `[?modify]` returns a new document; the input is observably unchanged. On a naïve implementation this would mean every `modify` allocates a full copy of the document — prohibitive for the 10 MB documents that CX actively targets, and impossible at the gate-15 streaming-throughput envelope (200+ MB/s).
The resolution is structural sharing — the same idiom that powers Clojure's persistent data structures, Haskell's lenses, and Scala's case-class `.copy`. Most of the new document is the same bytes as the old document; only the path from root to the changed node is freshly allocated.
The spine-copy algorithm
For each focus match `m` in the input document `D`: walk from `D.root` down to `m`, collecting the path `p₀ → p₁ → … → pₖ`; allocate fresh element headers `p₀'`, `p₁'`, …, `pₖ'` (each `pᵢ'` copies `pᵢ`'s attributes and children, with one slot reassigned to point at `pᵢ₊₁'`); apply the action to `pₖ'`; return `p₀'` as the new document root.
input doc: output doc (after :set on root/foo/@bar):
root [a] root' [a] ← new
├─ foo [b] ├─ foo' [b'] ← new
│ ├─ bar=1 ─┐ │ ├─ bar=2 ← new (set)
│ ├─ baz ─┼─── shared ──┤ ├─ baz ← same ref
│ └─ children ┘ │ └─ children ← same ref
└─ qux └─ qux ← same ref
"Same ref" is literal — same pointer in memory. The two documents differ in identity (hashes differ, `equals` returns false), but share ~99% of their bytes in RAM. Boehm GC tracks the shared subtrees and reclaims them when both documents become unreachable.
Cost model
Allocation is O(depth), not O(document_size). For `N` matches at average depth `d̄` sharing a common prefix of depth `c`, total cost is `O(N(d̄ - c) + c)`. The shared-prefix optimization is automatic — the spine-copy algorithm only allocates each new header once.
Per-action heap: `set` and `set-attr` allocate `O(d)` headers; `delete` and the insert family `O(d) + O(n)` for the children array; `replace`, `using fn`, and `rename` `O(d)` headers plus their value-specific payload.
The identity invariant
The input document `D` is observably unchanged after any `[?modify]`. This holds because no Layer-1 method exposes a mutating Node accessor. The C ABI has no `cx_node_set_attribute` and never will.
[?lib 'cx-stdlib/format']
[?lib 'cx-stdlib/hash']
[db [user status='new']]
[?let [= $doc [$first //db]]
[= $before-hash [$hash:sha256 [$format:canonical $doc]]]
[= $modified [?modify $doc //user/@status [set 'verified']]]
[= $after-hash [$hash:sha256 [$format:canonical $doc]]]
[identity-check equal=[= $before-hash $after-hash]]]
Concurrent reads are free
Because no Node is ever mutated, multiple threads can read the same Node concurrently without locks. `cx_thread_register` initialises per-thread GC state but is not synchronization for the data graph. Writes are not concurrent in the locking sense — each `[?modify]` returns a new Doc; the input Doc isn't touched. Cross-thread structural sharing is automatic via pointer equality.
Why not copy-on-write?
A COW design — every Node has a refcount; bump on write; copy on conflict — was considered and rejected: the current Node layout has no refcount field (adding one bloats every Node header by 4-8 bytes, 10-25% of the header); COW makes write paths conditional, which branches the hot path inside the evaluator (spine-copy is unconditional); the shared-subtree contract is already established by Boehm GC tracing — COW would be a parallel mechanism doing the same job worse.
Wasm three-tier rendering and respin status
The wasm story has two pieces: the long-term distribution story (one libcx.wasm replacing the per-triple .so / .dylib matrix) and the three-tier diagram-rendering split (gate 17). This section frames both.
Why WebAssembly
CX ships today as a native C ABI library — libcx.dylib on macOS, libcx.so on Linux — that every binding loads through its own foreign-function machinery. It works and it is fast, but it carries two ongoing costs: every supported triple needs a binary; every binding carries its own FFI plumbing. A WebAssembly target collapses both.
The long-term distribution target is a single `libcx.wasm` artifact loaded by each binding through its host runtime — wasmtime, wasmer, the browser, Node, V8. The function names, the binary wire protocol, and the per-language Document API all stay the same. The boundary moves; the contract stays.
Three-tier diagram rendering
The reference renderer (gate 12) makes every well-formed CX program renderable to SVG, PNG, or Mermaid text per `spec/code.md` §10.1.2. Three rendering tiers land at different capability levels — the playground commits to tier 1, the editor and CI integration story commits to tier 2, and tier 3 is reserved for a future revision.
- **Tier 1** — Mermaid text via `libcx.wasm`, rendered to inline SVG by Mermaid.js in the browser. Pure browser, no server. Powers the playground Visualize affordance per gate 17. - **Tier 2** — Graphviz SVG/PNG via an HTTP service composed in CX with `[?http-service on=http]` wrapping `cx diagram`, which shells to `dot -Tsvg`/`-Tpng`. The supported integration story for editor plugins and docs CI when source text is too large or the visual richer than Mermaid handles. Composes cleanly with `[?service]` in ~30 lines of CX. - **Tier 3** — bundled wasm-graphviz (e.g., `@hpcc-js/wasm`) running `dot` layout entirely in the browser, ~1 MB additional bundle. Planned once tier 1 + tier 2 are exercised in the wild.
Diagrams round-trip per gate 9 — every renderer output carries the original program bytes as metadata (Mermaid leading `%%cx:<base64>%%` comment, SVG `<metadata><cx:source>` block, PNG `tEXt` chunk keyed `cx-source`), and `reverse_parse_diagram` recovers them verbatim. The visual layer is a projection; the CX source is the source of truth.
Wasm respin status
`libcx-wasm` is shipped (respun from the original v0.7.5 cut to track the renamed code surface). [; version-literal-ok ] V source compiled through V-emit-C and then emscripten, a focused C ABI subset for the playground-load-bearing surface (`cx_code_eval`, `cx_to_cx`, `cx_to_json`, `cx_to_xml`, `cx_canonical`, `cx_hash`, plus the format converters and `cx_free` / `cx_version` / `cx_features` introspection), and a hand-written JavaScript wrapper at `scripts/wasm/cxlib.js` that exposes the surface via linear-memory marshalling. Build with `make build-wasm`; artifacts land in `dist/wasm/{libcx.wasm,libcx.js,cxlib.js}`.
Regex-using filters raise an explicit `cx-err:CXER0100` (regex-unavailable-in-wasm); RE2 is C++ and not linked into the WASM build at this tag. A JavaScript `RegExp` host shim is filed as a follow-up if demand materialises. The native `cx` binary remains the supported path for regex-heavy code.
Performance numbers — parse, emit, eval, FFI
Throughput on a single core. Bench setup: M2 Pro, libcx built with `-prod`, 1 MB fixture per format. The harness lives at `vcx/tests/runners/streaming_bench.v` and `vcx/tests/runners/eval_features_bench.v`; run `make bench` to refresh `target/bench.json`. The cross-binding parity check at `scripts/run_bench_json.py` compares per-binding times against the V baseline.
Parse
Parse throughput, indicative ranges (refresh each release): - **CX** — ~250 MB/s (the canonical-form parser is slower than dialect-specific parsers because it handles type-strict scalars, sized integer ranges, and atom recognition in one pass). - **JSON** — ~400 MB/s (simdjson-class on small documents). - **XML** — ~180 MB/s (entity-reference and namespace resolution cost). - **YAML** — ~85 MB/s (implicit-typing rules and folding eat throughput).
Emit
Emit throughput follows parse — same order of magnitude per format. The canonical-form emitter pays an extra normalisation pass and runs around 60% of the regular emit.
Eval
The CX code streaming evaluator runs at ~340 MB/s on the comparable bench corpus — about 17% under the comparable JSON benchmark on the same workload. The v0.7.0 target was 300 MB/s; that line has been [; version-literal-ok ] crossed.
FFI overhead
Per-call FFI overhead is in the low microseconds. Python ctypes adds about 35 µs per parse-and-emit round-trip on a 1 KB input; Go cgo is similar; Rust `bindgen` is closer to 1 µs. See the per-binding pages (§7.10.x) for the exact wrappers.