Identity

Lossless vs strict canonical (recap)

Two canonical forms (see canonical-strict + canonical-lossless in surfaces for the surface-level treatment):

• Lossless canonical (cx fmt) — preserves comments, anchors, aliases, presentation. Idempotent. Used for source-file formatting and diff stability.
• Strict canonical (cx canonical / cx hash) — strips comments and directives, expands anchors and merges, normalizes datetime offsets to UTC, renames declared ids to a deterministic scheme. Two docs with the same data hash identically. Used for content addressing.

The hash is taken over strict canonical bytes, not over lossless. Attribute order is source order — it is part of the data and part of the canonical bytes. Canonicalization never reorders attributes, so [user email='a' id=1] and [user id=1 email='a'] are two different documents with two different hashes. Order carries meaning in CX; nothing in either canonical tier sorts it away.

          [$cx:canonical [$cx:parse "[user
  [; regrettably noisy comment ]
  email='ada@example.com'
  id=42
  active=true]"]]
        

          $ cx prog.cx
'[user email=ada@example.com id=42 active=true]\n'
        

Canonicalization rules (strict):

• Attribute order: source insertion order, never sorted. The only fixed ordering is the element-meta slots themselves: anchor (&), merge (*), id (#), type annotation, then the attributes.
• String quoting: bare when the value is bare-eligible and would not re-parse as a typed literal; single quotes when quoting is required; double quotes only to avoid escaping an apostrophe. Escapes are minimal — only what the parser requires.
• Whitespace: no insignificant whitespace; quoted body text is preserved verbatim; structural newlines only where the layout rules require them.
• Numeric form: integers as plain decimal digits (hex converts to decimal in strict form); floats in the shortest representation that round-trips to the same value (Ryū shortest-form).
• Boolean form: true / false.
• Date/datetime: strict canonical normalizes the offset to UTC (2026-05-22T10:00:00-04:00 becomes 2026-05-22T14:00:00Z). The lossless form (cx fmt) preserves the offset as authored.
• Anchors expanded: &base / *base structure is expanded and removed — the canonical output contains only the resolved data, no anchor names.
• Ids renumbered: #name declarations become id-1, id-2, … in document order, and every @name reference is rewritten to match — so two documents differing only in id spelling share one hash.

Cross-format identity (hash stable through projections)

The hash is over CX strict canonical bytes — NOT over the projection (JSON / YAML / XML / etc.). This produces a key property: a document's hash is stable through lossless round-trip projections.

          [$eq [$cx:hash [users [user id=1 email='ada@example.com']]] [$cx:hash [$cx:from-format [$cx:to-format [users [user id=1 email='ada@example.com']] "xml"] "xml"]]]
        

          $ cx prog.cx
true
        

The property holds for the lossless projections: XML, and the --lossless JSON / YAML envelope encoding (see roundtrip). The default (idiomatic) JSON / YAML projections and the TOML projection are typed-lossy — the re-imported document is a different document with a different hash. Lossy projections (Markdown, CSV/TSV) drop information outright. CX makes the lossy-ness explicit; see roundtrip for the full matrix.

Practical use: in a content-addressable store, you can accept incoming data in a lossless format, convert to CX, hash, dedupe — knowing that two clients sending the same data through lossless projections will produce the same key.

Stability across bindings + machines

The hash function is stable across bindings, OSes, and architectures. All four bindings produce byte-identical hash strings for byte-identical input. The guarantee extends to architecture (x86-64, ARM64, little-endian, big-endian) and OS (macOS, Linux, Windows, BSD). Verified by conformance/binding_api.cxd gate 28.6.

Determinism details:

• Float formatting: the canonical shortest-form produces byte-identical output regardless of host C library (printf("%g") is NOT used; CX uses the Ryū shortest round-trip formatter).
• Endianness: the canonical bytes are UTF-8 text, no multi-byte numeric encoding in the canonical form itself.
• Unicode normalization: identifiers and string content are normalized to NFC before canonicalization.
• Locale-independence: the one place canonical form sorts anything — map keys — sorts by the Unicode order of the canonical key serialization, never by locale collation (no LC_COLLATE influence). Attributes are never sorted at all.

Truncation safety (short ids)

SHA-256 is 256 bits (64 hex chars). For human-facing identifiers you can truncate — a short prefix is usually enough to tell a handful of documents apart in a UI, a URL, or a log line. The trade-off is the birthday bound: the probability of a collision grows with the square of the collection size, so a prefix that is comfortably unique for dozens of items becomes risky for millions. Truncation is a display convenience, not an identity: store, compare, and sign with the full 64-hex digest, and treat any truncated form as resolvable back to it (the way git resolves short commit ids).

Use cases — cache keys, dedup, content stores

Cache key: cx hash request.cx is a stable cache key for an expensive computation parameterized on the document. Same input → same key → memoization works.

Deduplication: in a document store, hash on ingestion; store the canonical bytes once, point all duplicates at the same blob.

Content-addressable storage: the hash IS the address. Git, IPFS, Nix all use the same model. CX hashes interop directly:

• Git LFS: point cx hash-derived blob ids at the CX file's strict-canonical bytes.
• IPFS: wrap the canonical bytes in a CIDv1 dag-pb envelope or use raw codec; the SHA-256 maps to the CID's multihash.
• Nix: identical store-path-shape (SHA-256-named blob with metadata sidecar).
• S3 / object storage: content-MD5 header is too weak; use x-amz-meta-cx-sha256 for stronger integrity.

Signing: sign the strict canonical bytes; verify by re-canonicalizing on receipt. The signature is robust against benign reformatting (whitespace, quoting, comments) because those don't affect strict canonical form.

Buffer envelope

An ast-bin payload is deliberately spare: a 4-byte size header, then the payload. There is no magic number, no capability-bit block, no varints, and no trailer — feature negotiation happens out-of-band via cx_features (see ast-bin-capabilities), not inside every buffer. Layout per spec/03-approved/core/ast-bin.md §1–§3:

• Size header (4 bytes): u32 little-endian byte length of the payload (excluding the header itself). The buffer is opaque binary, not a null-terminated string.
• Version byte (1 byte): producers emit the lowest version that carries the document — 6 for the common case, 8 when a Path / Match / Modify node is present, 9 when an element carries a [table[…]] payload. Decoders reject buffers versioned above what they support.
• Prolog: u16 little-endian prolog node count, then that many nodes.
• Elements: u16 little-endian element count, then that many nodes. Each node is [u8 node-tag] followed by its per-kind payload, recursively — a preorder node-tag stream.

Strings on the wire are [u32 LE byte length][UTF-8 bytes] — fixed-width length prefixes, no null terminators, no varints. Optional strings carry a one-byte present/absent flag first.

Capability bits (full table)

Capability bits are the forward-compat mechanism. cx_features reports a 64-bit mask of what the loaded libcx implements; bindings check it on load and refuse to use features the library does not advertise. When a newer libcx adds a feature, it claims a new bit; bits are never re-assigned (retired bits stay reserved). The full table per spec/03-approved/core/abi.md §3:

Bits 42–63 are reserved (always zero). The cap-bit layout is fixed; bits are never re-assigned — retired features (bits 8, 13, 17) leave their bits permanently reserved.

Version negotiation + forward-compat

Negotiation is out-of-band and load-time, not per-buffer. cx_features() returns a NUL-terminated lowercase hex string encoding the 64-bit capability mask; a binding calls it once on load, caches the mask, and refuses to use features the loaded library does not advertise.

• Producers never lose data silently: a producer asked to encode a value whose kind requires an unadvertised bit either implements the kind or refuses to encode the document (emitters without the v8 wire slot reject Path values with CXER0290). There is no downgrade-and-drop path.
• Decoders fail loud: a buffer whose version byte exceeds the decoder's maximum is rejected before any node is read, and reserved or unknown node tags are rejected rather than skipped-and-guessed.
• Older buffers decode forward: a newer decoder reads any lower-versioned buffer; absent extensions default to their zero values.

Self-describing mode

Every value is tagged with its kind. Comparable to JSON or BSON in flexibility. Used when the schema is not known to both sides — e.g., wire-formatted between two CX bindings that don't share schemas, or for ad-hoc tooling.

Tag bytes are one byte, assigned in ranges: 0x00 null, 0x01 false, 0x02 true; 0x10–0x1F integers (width-tagged — 0x10 int8 through 0x17 uint64, 0x18 bigint); 0x20–0x2F floats and decimals; 0x30–0x3F string, bytes, and temporal scalars; 0x40–0x4F array containers; 0x50–0x5F map containers; 0x60–0x6F table containers. Canonical writers select the narrowest integer tag that preserves the value. Full tag table in spec/03-approved/core/data-bin.md §3.

Schema-driven mode (cap bit 24)

Per-value type tags are omitted wherever the schema declares the value's type; the reader walks the schema in lockstep with the data. Wire size comparable to Protobuf or Avro. Signaled by capability bit 24 and by header flag bit 1 in the payload itself.

Negotiation: when flag bit 1 is set, a schema reference sits in the header, between the fixed 12 bytes and the root value — most commonly tag 0x10 followed by 32 bytes: the SHA-256 of the schema's strict canonical text. (An inline-schema form and a hash-plus-name-hint form also exist.) Readers resolve the schema by content hash and fail closed if it is unavailable or its hash mismatches — never silently corrupted bytes. Tag omission is per-field: declared fields drop their tags, undeclared fields fall back to self-describing form, so both modes coexist in one document.

Chunked tables (§3.11)

Tables that exceed memory get a dedicated wire form — the chunked table, tag 0x63 per spec/03-approved/core/data-bin.md §3.11. Layout:

• Column specs, once: column count, then per-column name + type — the same col-spec shape as the in-memory table form.
• Row groups: zero or more, each uvarint(body-byte-length) + a one-byte body tag (0x01 plain, 0x90 zstd-compressed) + the body — row count plus column-major payloads for that group's rows.
• End marker: a single 0x00 closes the table.

The byte-length prefix on every row group is the whole skip-ahead mechanism: a reader can jump group to group without decoding bodies it doesn't need, and memory is bounded by the largest single row group. There are no per-chunk statistics, no manifest, and no trailer — predicate pushdown is deliberately ceded to Parquet via the Arrow bridge (see analytics).

Page-compression wrapper (§3.12)

A row-group body may be wrapped in compression (the 0x90 body tag, per spec/03-approved/core/data-bin.md §3.12; capability bit 22). Exactly one codec is defined: zstd v1 (codec-id 0x01); readers reject any other codec id with error D005. The wrapper applies only at the row-group body position — never at value level or inside a column payload.

The hash is taken over the uncompressed bytes: compression level, dictionary use, and zstd block boundaries do not affect cx_hash. Storage/transport concerns and content identity stay orthogonal.

Document-unique scoping

An identity #name is unique within a single CX document. A duplicate #name declaration is refused when the document is read, with CXER0208 (E_DUPLICATE_ID) — the error names the id and the fact that it is declared on more than one element.

Identities do NOT cross document boundaries. Two separate CX files can each declare #ada without conflict; they refer to different elements in their respective documents.

IDREF resolution + cycles

A reference to an identity uses the @name sigil in attribute-value position: [post author-id=@ada …]. In body position, the reference form is a [ref @name] element. # declares; @ references — the two sigils are never interchangeable.

          [users
  [user #u-1 name=alice]
  [reviewer assigned-to=@u-1]
  [approver checked-by=@u-1]
]
        

          $ cx --from=cx --to=cx input.cx
[users
  [user #u-1 name=alice]
  [reviewer assigned-to=@u-1]
  [approver checked-by=@u-1]
]
        

Reference cycles (A references B references A) are not an error — the references are values, not container relationships; no infinite recursion. However, cyclic merge references (via the * sigil — see merge) ARE errors: a document whose anchors resolve through themselves has no finite expanded form, so canonicalization refuses it outright.

The ID/IDREF model is XML-compatible: round-trip through XML preserves the references (xml:id + IDREF-typed attributes). The semantic JSON / YAML / TOML projections emit an ordinary id field and a {"$ref": "…"} envelope — a one-way, human-readable projection per spec/03-approved/core/conversions.md §1.1.

Identity preservation across projections

Only CX and XML carry the ID/IDREF mechanism losslessly; the other projections flatten it into plain fields (spec/03-approved/core/conversions.md §1.1):

The $ref shape is operator-facing (humans recognize it as a reference); it is not machine-round-trip-authoritative. When identity must survive a pipeline, route it through CX or XML.

Ownership across the C ABI

The ABI is string-in / string-out: a conversion symbol takes a UTF-8 buffer, does its work, and returns a freshly-allocated buffer. There is no persistent document handle living inside libcx — no reference to release, no finalizer to register for ordinary conversions.

• Returned buffers are caller-owned: every string or byte buffer libcx returns is released with cx_free — never with the system free().
• A binding's Doc is host-owned: it holds the document in the host language's own memory (text and/or a binding-native AST decoded from ast-bin). Its lifetime is ordinary host memory management — the host's collector, reference count or scope drop reclaims it. Nothing in libcx needs to be notified.
• The exceptions are the streaming handles: the event reader (cx_events_open / cx_events_next / cx_events_close) and the streaming table reader / writer keep state across calls behind an opaque handle with an explicit close. Bindings wrap these in the host's resource idiom (context manager, defer, Drop).

Practical implications:

• Long-lived Docs: cost host memory only; libcx holds nothing on their behalf between calls.
• Multi-process sharing: ast-bin is the wire format — serialize on one side, deserialize on the other. There is no shared-memory Doc passing.

Nodes are owned values, not borrows

A Node selected from a Doc (via select_all, select, find_all, etc.) is an owned host value — a binding-native piece of tree, not a pointer into libcx memory. Nothing is borrowed across the FFI boundary, so there is no lifetime coupling to manage:

• Dropping the Doc does not invalidate Nodes already selected from it — each Node stands on its own in host memory.
• No cross-language GC interplay: the host language manages Docs and Nodes exactly like any other value it allocated itself.

Doc.bytes() output is owned: the canonical bytes returned from doc.bytes() are a freshly-allocated copy owned by the caller. Modifying or freeing it does not affect the Doc.

Doc.modify returns a NEW Doc: the input Doc is unchanged — value semantics, not aliasing. The two Docs lead independent lives; releasing one never affects the other.

Thread registration (cx_thread_register)

libcx uses Boehm GC, which requires that every thread that calls libcx APIs is registered with the GC. The C ABI exports cx_init() (call once per process at start) and cx_thread_register() (call per thread before first libcx call from that thread). Per spec/03-approved/core/abi.md §1.5.

Bindings handle this automatically at the FFI chokepoint — every Layer-1 method has a per-thread guard that registers the calling thread on first use. Host application code doesn't call cx_thread_register directly unless dropping below the binding layer to raw FFI.

Cap bit 26 advertises the thread-init handshake. It is required for bindings whose host runtime spawns OS threads outside V's control (Rust, C#, Java), and cheap and harmless everywhere else.

Practical implication: spawning a thread from host code and calling a CX binding method from it is safe — the binding registers the thread transparently. The only caveat: if you're calling libcx via raw FFI (not through a binding), you must call cx_thread_register() yourself before any other libcx call from that thread.

Canonical form recap

Canonical mode strips presentation — comments, anchors, whitespace — normalizes datetimes to UTC, renames ids to their deterministic document-order names, and emits a data-equivalent byte-identical form. Two documents with the same data emit to the same bytes; the difference between them is something a human added, not something the data carries. (Attribute order is data, not presentation — it is preserved and hashed.)

          [$eq [$cx:hash [$cx:parse "[a   1]"]] [$cx:hash [a 1]]]
        

          $ cx prog.cx
true
        

          [$cx:equal [a 1] [a 1]]
        

          $ cx prog.cx
true
        

Canonical form is the basis for content-addressed identity (cx hash — a tagged address, sha2-256: being the registry default), for byte-identical conformance tests across bindings, and for diff that ignores presentation.

Declared identifiers (#id, @ref)

Declared identity is what you write when you want one element to point at another by name. Declare with #; reference with @. IDs are unique within their scope; the scope is the document root by default.

          [doc
  [users
    [user #u-1 name=alice]
  ]
  [reviews
    [review target=@u-1 score=5]
  ]
]
        

          $ cx --from=cx --to=cx input.cx
[doc
  [users
    [user #u-1 name=alice]
  ]
  [reviews
    [review target=@u-1 score=5]
  ]
]
        

Anchors and merge targets

An anchor (& sigil) names a value once and lets later places refer to it without repeating it. A merge target (* sigil) folds the anchor referent into the current element. Anchors are anonymous handles; the strict canonical form expands every merge and removes the anchor structure entirely — no anchor name survives into the canonical bytes, so two documents that bind the same data with different anchor names hash to the same identity.

          [$cx:equal [$cx:parse "[x [a &n 1] [b [*n]]]"] [$cx:parse "[x [a 1] [b [a 1]]]"]]
        

          $ cx prog.cx
true
        

ast-bin quick reference

ast-bin is the binary serialization of the canonical form. The buffer envelope is [u32 LE size][payload bytes]; payload is the AST as a preorder node-tag stream whose node kinds are gated by capability bits (column-major layout belongs to data-bin's table payloads, not to ast-bin). Release the buffer with cx_free; never call system free(). Capability bit 36 advertises the PathNode wire format (ast_bin v8); cap bit 3 advertises symmetric AST round-trip. The full bit table is in spec/03-approved/core/abi.md §3.

data-bin quick reference (CXCol v1)

data-bin is the strict-canonical binary data format — format identifier CXCol v1. The magic is the five ASCII bytes CXCol (0x43 0x58 0x43 0x6F 0x6C); readers reject anything else outright. Modes: self-describing (default), schema-driven (cap bit 24 — header flag bit 1, schema referenced by the SHA-256 of its strict canonical text), chunked tables (spec/03-approved/core/data-bin.md §3.11, cap bit 21), and page-compressed (zstd v1, cap bit 22). Chunk boundaries are part of the canonical bytes, so chunked output is byte-stable across writers only at the canonical chunk size (2²⁰ rows per group); compressed row groups hash over their uncompressed bytes, so compression level never moves an address.