Analytics, columnar formats, and bridges

The analytics lane

CX has an analytics lane. The same brackets that carry a config file also carry a typed columnar table; the same canonical-bytes-plus-hash identity that anchors a single document also anchors a billion-row dataset; the same evaluator that templates a Markdown report also streams over a multi-gigabyte file in bounded memory. The lane has three load-bearing pieces, and this section is about how they compose.

The three pieces are: a **logical data shape** (the `:table` type tag, with typed columns and nullable cells); a **physical wire format** (CXCol, the column-oriented streaming binary form, tag `0x63` in [`spec/core/data-bin.md`](../../spec/core/data-bin.md) §3.11); and **bridges to the broader ecosystem** (Apache Arrow C Data Interface for zero-copy interop, Apache Parquet for cross-tool storage).

  • **Logical**: `[orders :table :columns (...) (...) ...]` — see table-shape. Authored in CX text, parses into the same AST as any other CX element, validates against `.cxs` schemas, hashes via SHA-256 over canonical bytes.
  • **Physical**: CXCol — the column-oriented streaming binary form. Row-group chunked, optionally zstd-compressed per chunk, optionally schema-driven for tag-density. See cxcol.
  • **Bridges**: Arrow C Data Interface (zero-copy in-process handoff) and Parquet (cross-ecosystem storage). See arrow-bridge and parquet-bridge.
  • **Streaming**: the CX streaming evaluator (see [`spec/streaming.md`](../../spec/streaming.md)) emits `StartTable` / `RowGroup` / `EndTable` events over CXCol data, so directives like `[?for]` and `[?modify]` can operate on tables that don't fit in memory.
  • **Identity**: CXCol files have a stable SHA-256 over their canonical (uncompressed) bytes. Compression level does not change the hash. Chunk boundaries do — see cxcol-identity for the rule.

The rest of this section walks each piece. The normative spec documents are [`spec/core/data-bin.md`](../../spec/core/data-bin.md) §3.10-3.13 (wire format) and [`spec/misc/table-api.md`](../../spec/misc/table-api.md) (host binding API). This guide is the working developer summary.

**Naming.** `CXCol` is the column-oriented streaming format name. The `CXDB` / `CXDatabase` namespace is reserved for a future query-engine product and does not refer to the file format. Throughout this guide, CXCol is the format; `.cxcol` is the file extension; the `cxcol_*` prefix appears on host-binding identifiers.

The :table data shape

The `:table` type tag declares a typed columnar table at the logical CX layer. It is a CX element like any other — same brackets, same parse rules, same canonical-bytes contract — but the parser recognizes the `:table` body shape and produces a `Table` value rather than a generic element.

          [orders :table
           :columns (id :int, customer :string, amount :float, ts :datetime)
           (1, 'Ada',   42.50,  2026-05-22T10:00:00Z)
           (2, 'Grace', 17.25,  2026-05-22T10:15:00Z)
           (3, 'Linus', 99.00,  2026-05-22T11:30:00Z)]

The `:columns` attribute carries the column spec: a parenthesized sequence of `name :type` pairs. The body is a sequence of row tuples, each tuple matching the column count. Tuple values must match the declared column types (or be `null` for nullable columns); cell-type mismatches are a parse error (`E_TABLE_CELL_TYPE`).

Supported column types match the CX scalar type system: `:int` / `:i8..:i64` / `:u8..:u64` (sized integers), `:float` / `:f32` / `:f64` / `:f16` (floats), `:bool`, `:string`, `:bytes`, `:date`, `:datetime`, `:decimal(precision, scale)`, `:bigint`, plus collection columns (`arr[T]`, `map[K, V]`, `seq[T]`). The full type catalog with byte encodings is in [`spec/core/data-bin.md`](../../spec/core/data-bin.md) §3.10 + [`spec/misc/type-mapping.md`](../../spec/misc/type-mapping.md).

Nullable columns

Nullable columns are declared by wrapping the type in `nullable`. A null cell is written as the literal `null`.

              [customers :table
             :columns (id :int,
                       name :string,
                       phone (nullable :string))
             (1, 'Ada',   '555-1212')
             (2, 'Grace', null)
             (3, 'Linus', '555-9999')]

Nullable wire encoding is the def/rep-level pattern from [`spec/core/data-bin.md`](../../spec/core/data-bin.md) §3.10.5; the host binding's `Table.cell(r, c)` returns the host's null sentinel (`None` in Python, `nil` in Go, `Option::None` in Rust).

Dictionary encoding

Tables are dictionary-encoded by default for `:string` columns with low cardinality. The encoder builds a per-chunk dictionary of distinct values; the column buffer stores dictionary indices instead of raw strings. Decoders rebuild the strings transparently; the host-binding API never exposes the indices.

Dictionary encoding is automatic, not a user-visible flag. The threshold (≥ 4× compression vs. raw on the chunk) lives in [`spec/core/data-bin.md`](../../spec/core/data-bin.md) §3.10.4. CXPath `=` and `[?match]` value comparison operate on the decoded string, so user code is dictionary-agnostic.

Schema validation

Tables validate against `.cxs` schemas the same way ordinary elements do. A schema describes the column spec; the validator enforces row-count, type-conformance, and nullability per cell. See [[data#schema]] for the schema language and [`spec/schema.md`](../../spec/schema.md) §6 for table-specific rules.

              [# orders.cxs — schema describing the orders table shape #]
           [schema
             [table :name orders
               [column :name id       :type int    :required true]
               [column :name customer :type string :required true]
               [column :name amount   :type float  :required true]
               [column :name ts       :type datetime]]]

The Table host-binding API

Bindings expose a `Table` value type when `loads` encounters a `:table` block. Methods match across bindings up to host-idiom differences; the canonical surface is documented in [`spec/misc/table-api.md`](../../spec/misc/table-api.md).

              import cxlib
           doc = cxlib.loads(open('orders.cx').read())
           t   = doc['orders']                     # Table instance

           print(t.cols)         # ['id', 'customer', 'amount', 'ts']
           print(t.row_count)    # 3
           print(t.row(0))       # OrderedDict([('id', 1), ('customer', 'Ada'), ...])
           print(t.column('amount'))  # [42.50, 17.25, 99.00]

           # Slicing / projection
           sub = t.head(2).select(['customer', 'amount'])
           print(sub.to_cx())

The 17-member canonical surface (`cols` / `types` / `row` / `column` / `cell` / `slice` / `head` / `tail` / `select` / iteration + adapters) is described in [`spec/misc/table-api.md`](../../spec/misc/table-api.md) §3. The same shape exists in V, Go, Rust, and TypeScript with the naming conventions of each host.

CXCol — column-oriented streaming format

CXCol is the physical wire form for tabular data: column-oriented, row-group-chunked, content-addressable, and streamable in bounded memory. The data-bin tag is `0x63`; the file extension is `.cxcol`. The normative byte-level layout is in [`spec/core/data-bin.md`](../../spec/core/data-bin.md) §3.11.

The format is a single sequential pass: header, schema preamble, then a stream of row groups, then a terminator. Writers emit one row group at a time and can produce a billion-row table in memory bounded by the largest row group. Readers pull row groups one at a time and can skip groups by their length prefix without decoding them.

Wire layout

A CXCol stream is:

  • **12-byte header.** Magic `CXDB`, format version, capability bits, header flags. Header flag bit 0 reserved; header flag bit 1 signals schema-driven mode (see cxcol-schema-driven).
  • **Schema preamble.** The column spec: count, then one `(name, type_tag, nullable_flag)` triple per column. Encoded once at the head — every row group in the file conforms to this schema.
  • **Row-group sequence.** Each group is `uvarint(body_byte_len) <body-tag> <body>`. `body-tag` `0x01` = plain (uncompressed column buffers); `0x90` = compressed with codec id (`0x01` = zstd v1).
  • **Per-chunk manifest** (inside each group). Row count, optional per-column min/max/null-count statistics. The statistics are not yet used by libcx itself but are reserved space the Parquet bridge populates on emit.
  • **Terminator.** A single `0x00` byte (zero-length row group) closes the stream.

The 2²⁰ (1,048,576) rows per group is the canonical chunk size — `cx canonical` enforces it on a CXCol input, and writers targeting cross-system hash invariance should use it. Different chunk sizes produce valid CXCol files but with different SHA-256 hashes.

Content-addressable identity

A CXCol file's SHA-256 hash is computed over its **canonical uncompressed bytes** — the bytes you would see if every `0x90` page-compression wrapper were unwrapped to its `0x01` plain form. Compression is a transport / storage choice; identity is a content choice; the format keeps them orthogonal.

  • **Same data + same chunking + same schema → same hash**, regardless of zstd level or whether compression was applied at all.
  • **Different chunk size → different hash.** Chunk boundaries are part of the canonical form. The 2²⁰-rows-per-group convention is the cross-writer interoperability rule.
  • **Hash composes with hash.** The same SHA-256 primitive that hashes a CX document hashes a CXCol file — tooling pipelines (caching, dedup, signing, content-addressable storage) treat both uniformly.
              import cxlib
           # Two CXCol writers, same source data, different compression
           cxlib.write_cxcol('a.cxcol', table, compress='zstd:3')
           cxlib.write_cxcol('b.cxcol', table, compress='zstd:9')

           # Files differ on disk:
           open('a.cxcol', 'rb').read() == open('b.cxcol', 'rb').read()  # False

           # But canonical hashes are identical:
           cxlib.cxcol_hash('a.cxcol') == cxlib.cxcol_hash('b.cxcol')    # True

Streaming reader / writer

Bindings expose a handle-based streaming API mirroring the C ABI in [`spec/core/data-bin.md`](../../spec/core/data-bin.md) §3.11.6 (10 symbols under capability bit 21). The Python / Go / Rust / V shapes follow each host's resource-management idiom.

              import cxlib

           # Reader — pulls one row group at a time
           with cxlib.cxcol_reader('big.cxcol') as r:
               print(r.schema)            # column spec
               for group in r:            # iterator of row-groups
                   process(group)         # group is a Table

           # Writer — pushes one row group at a time
           with cxlib.cxcol_writer('out.cxcol', schema=cols) as w:
               for chunk in source_of_chunks():
                   w.emit(chunk)          # bounded memory per chunk
            
              package main

           import "cxlib"

           func main() {
               r, _ := cxlib.CxcolReaderOpen("big.cxcol")
               defer r.Close()
               for {
                   g, err := r.Next()
                   if err == cxlib.EOF { break }
                   process(g)
               }
           }
            
              use cxlib::cxcol::Reader;

           let mut r = Reader::open("big.cxcol")?;
           while let Some(group) = r.next()? {
               process(group);
           }
           // r dropped → reader closed via Drop

Memory budget is one row group in flight per stage in the pipeline. A reader chained into a CXPath filter chained into a Parquet writer holds three row groups at once, total — regardless of the table's logical row count. See streaming for the directive-level view.

Schema-driven mode

When the writer knows both sides of the wire have access to the schema, header flag bit 1 enables schema-driven mode: per- value type tags are omitted for columns the schema declares; reader and writer walk the schema in lockstep with the data. Wire density approaches Parquet's typed encoding.

Schema-driven mode is opt-in (`cxlib.write_cxcol(..., schema_driven=True)`); having a schema in scope does not silently promote the encoder. The schema reference forms (`0x10` content-hash, `0x11` inline, `0x12` content-hash + name hint) are documented in [`spec/core/data-bin.md`](../../spec/core/data-bin.md) §3.13.

Schema-driven mode requires the reader to retrieve the schema by its content hash. Local lookup is the default; the callback-shaped content-store ABI is a planned follow-up.

Arrow bridge — zero-copy interop

CXCol bridges to Apache Arrow via the Arrow **C Data Interface** — a struct-based ABI that lets two libraries share columnar buffers without copying them. The CX side is `libcx_arrow`, a separate dynamic library that depends on libcx + the Arrow C-Data ABI. Core libcx is Arrow-free; capability bit 23 advertises Arrow when libcx_arrow is loaded.

The bridge exports four C ABI symbols: `cx_arrow_export_open` / `cx_arrow_export_open_fd` for CXCol → Arrow, and `cx_arrow_import_to_data_bin` / `cx_arrow_import_to_data_bin_fd` for Arrow → CXCol. Bindings wrap them as `to_arrow()` / `from_arrow()` on the Table value type.

Zero-copy handoff

The Arrow C Data Interface uses two structs: `ArrowSchema` (column types + names) and `ArrowArray` (per-batch buffers, including null bitmap, offsets, and values). CX populates these structs to point at its own column buffers; consumers (DuckDB, Polars, PyArrow, etc.) read from the same memory pages CX wrote.

Ownership crosses the boundary explicitly. The exporter populates an `ArrowArrayStream` with a `release` callback; the consumer calls `release` when done. CX's release callback frees the underlying CXCol buffers. There is no GC interplay — bindings hold the Arrow stream as an opaque handle until they're done with it.

              import cxlib, pyarrow as pa

           # Load CXCol — buffers held by libcx_arrow
           t = cxlib.read_cxcol('orders.cxcol')

           # Zero-copy export to Arrow: no buffer copy, just pointer handoff
           reader = t.to_arrow_stream()          # ArrowArrayStream
           pa_table = pa.Table.from_batches(reader)

           # pa_table now references the same memory as t.
           # Don't free t (or call cxlib release) until pa_table is done.
            
              use cxlib::cxcol;
           use arrow::record_batch::RecordBatchReader;

           let t = cxcol::read("orders.cxcol")?;
           let mut reader: Box = t.into_arrow_stream()?;
           while let Some(batch) = reader.next() {
               let batch = batch?;
               // batch.column(0) points at the same memory as t's first column
               process(batch);
           }

Type mapping

The CX ↔ Arrow type map is normative. The high-level summary:

  • `:int` / `:i64` ↔ `Int64`; `:i32` ↔ `Int32`; sized variants (`:i8..:u64`) map to corresponding Arrow sized integer types.
  • `:float` / `:f64` ↔ `Float64`; `:f32` ↔ `Float32`; `:f16` ↔ `Float16` (Arrow ≥ 1.0).
  • `:bool` ↔ `Boolean` (bit-packed in Arrow).
  • `:string` ↔ `Utf8` (or `LargeUtf8` for columns whose total byte length exceeds 2 GB).
  • `:bytes` ↔ `Binary` (or `LargeBinary` similarly).
  • `:date` ↔ `Date32` (days since 1970-01-01, proleptic Gregorian).
  • `:datetime` ↔ `Timestamp(Nanoseconds, UTC)`; offsets round-trip via the `tz` field when present.
  • `:decimal(p, s)` ↔ `Decimal128` (p ≤ 38) or `Decimal256` (p ≤ 76).
  • `arr[T]` ↔ `List<T>`; `map[K, V]` ↔ `Map<K, V>`; `seq[T]` ↔ `List<T>` (sequence flatten per CXDM §1.2).
  • `nullable T` → Arrow's standard nullable variant (validity bitmap).

Current column-type coverage in `libcx_arrow` is 9 of 10 types (int / i8 / i16 / i32 / float / bool / string / date / bytes). `:datetime`, `:decimal`, and dictionary / extension columns remain deferred to a future libcx_arrow expansion; the wire surface accepts them but the C-Data bridge errors with `arrow-014-unsupported-type-deferred-error`.

Arrow IPC stream format

Beyond zero-copy in-process handoff, CXCol can emit Arrow IPC streams for cross-process or cross-host transport. The IPC stream is Arrow's own serialization (schema header + record batches + EOF), distinct from the C Data Interface (in-memory structs).

              import cxlib, pyarrow as pa

           # Emit Arrow IPC stream — bytes you can write to a file or socket
           with open('orders.arrow', 'wb') as f:
               for batch in cxlib.read_cxcol('orders.cxcol').to_arrow_ipc():
                   f.write(batch)

The `cx table dump --to=arrow` CLI subcommand (`spec/cx_table_cli.md`) writes the IPC stream form by default; `--arrow-format=file` opts into the file-format variant.

Arrow Flight (planned)

Arrow Flight is Arrow's gRPC-based RPC framing for cross- process record batches. It is the natural next layer above the IPC stream — a Flight server publishes batches by descriptor; a Flight client subscribes. CX integration is **planned**: no Flight surface ships currently; the tooling-layer story will be specified when it lands.

For cross-process CXCol transport currently, use Arrow IPC over your existing transport (HTTP, pipe, socket). The `[?http-client]` directive can carry Arrow IPC streams as binary bodies.

Parquet bridge

CXCol bridges to Apache Parquet via Arrow: CXCol → Arrow stream → Parquet file. Core libcx has **no direct Parquet C++ dependency** — adding one would couple every binding to a heavyweight build-time toolchain. The Parquet adapter is a binding-specific library (PyArrow for Python, arrow-rs for Rust, parquet-go for Go, etc.) chained off the Arrow bridge.

Writing Parquet

`Table.write_parquet(path, ...)` walks CXCol → Arrow record batches → Parquet writer. The Parquet writer is the host binding's choice; the CXCol side just provides the Arrow stream.

              import cxlib

           t = cxlib.read_cxcol('orders.cxcol')
           t.write_parquet(
               'orders.parquet',
               compression='snappy',     # Parquet ecosystem default
               row_group_size=1_048_576, # match CXCol canonical chunk
               use_dictionary=True,      # dictionary encoding (per-col)
               write_statistics=True,    # min/max/null per row-group
           )

Defaults are the Parquet-ecosystem defaults (Snappy, not zstd) — output is meant to be consumed by Parquet readers that assume Snappy. Zstd is CX's default for CXCol's `0x90` page compression; the two formats use their own ecosystem defaults to minimize friction with the consumers of each.

              use cxlib::cxcol;
           use parquet::file::writer::*;
           use parquet::basic::Compression;

           let t = cxcol::read("orders.cxcol")?;
           let mut writer = SerializedFileWriter::new(
               std::fs::File::create("orders.parquet")?,
               t.arrow_schema()?,
               Compression::SNAPPY.into(),
           )?;
           for batch in t.into_arrow_stream()? {
               writer.write(&batch?)?;
           }
           writer.close()?;

Reading Parquet

`read_parquet(path, ...)` is the symmetric inverse. The Parquet reader produces Arrow record batches; CX wraps them as a streaming Table.

              import cxlib

           # Whole-file read
           t = cxlib.read_parquet('orders.parquet')
           print(t.row_count)

           # Streaming read with projection and predicate
           for chunk in cxlib.read_parquet_chunks(
                   'orders.parquet',
                   columns=['id', 'amount'],     # only read these columns
                   filter='amount > 100'):        # predicate pushdown via stats
               process(chunk)

Predicate pushdown uses per-row-group min/max statistics emitted by the Parquet writer. A predicate like `amount > 100` against a row group whose `max(amount) = 50` skips the group's data pages entirely. Column projection only reads the requested column chunks. Together, selective queries on wide tables can reduce I/O by 10× or more — see performance.

Predicate pushdown depends on the Parquet writer having emitted statistics. The CX writer (5.5.1) does so by default (`write_statistics=True`). Files produced by other writers may not have statistics — the reader still works, but degrades to a full scan.

CX ↔ Parquet type mapping

The mapping is normative. Most types round-trip cleanly through Arrow; a small number of CX types have lossy edges that `--strict` mode rejects rather than silently coerces.

  • `:int` / `:i64` → `INT64`.
  • `:i32` → `INT32`; `:i16` / `:i8` → `INT32` with `INT(N, true)` logical type; unsigned similarly.
  • `:f64` / `:float` → `DOUBLE`; `:f32` → `FLOAT`; `:f16` → `FIXED_LEN_BYTE_ARRAY(2)` / `FLOAT16` (the latter is non-standard pre-Parquet 1.13).
  • `:bool` → `BOOLEAN`.
  • `:string` → `BYTE_ARRAY` with `STRING` logical type (UTF-8).
  • `:bytes` → `BYTE_ARRAY`.
  • `:date` → `INT32` with `DATE` logical type.
  • `:datetime` (UTC) → `INT64` with `TIMESTAMP(NANOS, isAdjustedToUTC=true)`.
  • `:decimal(p, s)` → `DECIMAL(p, s)` (Parquet caps at 256-bit precision).
  • `:bigint` → `Decimal256(38, 0)` with diagnostic; `--strict` rejects.
  • `arr[T]` → `LIST<T-projection>`.
  • `map[K, V]` → `MAP<K, V>` (string-keyed).
  • `seq[T]` → `LIST<T-projection>` after sequence flatten; `--strict` rejects (re-import produces `arr[T]`, not `seq[T]`).
  • `nullable T` → `OPTIONAL` repetition.

Round-trip identity

CX → Parquet → CX is not in general byte-identical. Parquet is a presentation-layer format: footer offsets shift, row-group sizes are writer-dependent, encoding choices (dictionary thresholds, page sizes, compression codec) are per-writer. Many byte-level Parquet files encode the same logical data. CXCol fills this niche precisely — it is the hashable origin; Parquet is the bridge.

**Logical** round-trip is normative: a CX table emitted as Parquet and re-imported via `cx table load` produces the same `:table` block with the same column names, types, row count, and cell values (modulo the documented lossy edges in 5.5.3). The `cx_eq` primitive operates on the logical level and reports equality after round-trip.

          import cxlib
           t1 = cxlib.loads(open('orders.cx').read())['orders']
           t1.write_parquet('orders.parquet')
           t2 = cxlib.read_parquet('orders.parquet')
           assert t1 == t2     # logical equality holds
           assert t1.to_cxcol_bytes() == t2.to_cxcol_bytes()
                               # canonical CXCol bytes match too

Metadata preservation

Parquet's `KeyValueMetadata` is a list of UTF-8 key-value pairs at the file footer. CXCol metadata slots — column comments, table-level annotations, the original schema content-hash — map to entries with the `cx_*` prefix (`cx_schema_hash`, `cx_schema_inline`, `cx_column_doc.<name>`). Readers that don't recognize the prefix ignore it; CX readers re-attach the metadata on import.

          import cxlib

           t = cxlib.read_cxcol('orders.cxcol')
           # Strict-schema mode embeds the CXCol schema as Parquet KV metadata
           t.write_parquet('orders.parquet', strict_schema=True)

           # Re-import preserves schema identity
           t2 = cxlib.read_parquet('orders.parquet')
           assert t.schema_hash() == t2.schema_hash()

Default writer mode does NOT embed the schema hash — Parquet consumers in other ecosystems wouldn't know what to do with it, and adding unrecognized metadata is a friction point. Use `--strict-schema` (CLI) / `strict_schema=True` (binding) when round-trip integrity matters.