Analytics, columnar formats, and bridges

Null cells

Any column can carry null cells. A null cell is written as the literal null — a bare token in cell position; an empty cell is not allowed in the wire form:

          [t [table[a b]]
  alice null
  bob 30
]
        

          $ cx --from=cx --to=cx input.cx
[t [table[a b]]
  alice null
  bob 30
]
        

On the wire, a column with null cells uses the nullable wrapper of spec/03-approved/core/data-bin.md §3.10.5: type code 0x80, then the inner type code, then a null bitmap of ceil(rows/8) bytes (bit set = null), then the packed payload of the non-null cells only. Each binding's table surface answers a null cell with its host's own "no value" form.

Collection cells

A cell is a value, so a cell may be a collection — an array or a map literal — and the table stays a table:

          [t [table[name tags]]
  alice [admin, user]
  bob [user]
]
        

          $ cx --from=cx --to=cx input.cx
[t [table[name tags]]
  alice ['admin', 'user']
  bob ['user']
]
        

          [t [table[name attrs]]
  alice {role: admin, age: 30}
  bob {role: user, age: 25}
]
        

          $ cx --from=cx --to=cx input.cx
[t [table[name attrs]]
  alice {role: admin, age: 30}
  bob {role: user, age: 25}
]
        

A table may also nest inside an element like any child, and an element that happens to be named table composes as an ordinary element — only the glued opener declares a table:

          [wrap [users [table[name age]] Ada 36]]
        

          $ cx --from=cx --to=cx input.cx
[wrap
  [users [table[name age]]
    Ada 36
  ]
]
        

Dictionary encoding

Dictionary encoding is an opt-in container form — tag 0x62 in spec/03-approved/core/data-bin.md §3.10.2. Each column's payload starts with a marker: plain (the §3.10.3 layout) or dictionary-encoded (distinct values in insertion order, then one varint index per row). Decoders rebuild the values transparently; no table surface exposes the indices.

Canonical writers emit the plain 0x60 form. A writer that opts into 0x62 uses a dictionary for a column only when the dictionary actually saves bytes — the canonical constraint of §3.10.2, evaluated per column — so categorical data (status, region, type) dictionary-encodes and high-cardinality data stays plain. CXPath = and [?match] compare the decoded value, so a program is dictionary-agnostic.

Schema validation

Tables validate against .cxs schemas the way ordinary elements do — the table block is an element, and the schema language of spec/03-approved/core/schema.md applies to it as to any other. See [[data#schema]] for the language. At the binary layer, a CXCol file can carry a schema reference that names its schema by content hash (see cxcol-schema-driven).

The table surface of a binding

Every binding exposes a table value for [table[…]] blocks — one surface, ratified in spec/03-approved/misc/table-api.md §3, spelled in each host's own naming: the cols / types / row_count / col_count properties; the row / column / col_at / cell / cell_by_name accessors; the slice / head / tail / select_cols projections; row iteration; and the to_cx / to_csv / to_json / to_data_bin / to_dict_list conversions. The four shipping bindings are on bindings; what a table method answers is what the corpus cases above answer, because the surface is a projection of the same value.

Wire layout

A CXCol stream is:

• 12-byte header. Magic CXCol (5 bytes, 0x43 0x58 0x43 0x6F 0x6C), version byte (0x01), flags byte, max_depth (uint32 LE), reserved byte. Flag bit 0 is endianness (1 = little-endian); flag bit 1 signals schema-driven mode (see cxcol-schema-driven); bits 2–7 are reserved and must be zero.
• Column spec. The chunked-table tag 0x63, then uvarint(col_count), then one col-spec per column — the column name as a tagged string plus its type code (the §3.10.1 shape). Declared once at the head — every row group in the file conforms to this spec.
• Row-group sequence. Each group is uvarint(body_byte_len) <body-tag> <body>. body-tag 0x01 = plain (uvarint(row_count) then column-major buffers); 0x90 = the zstd page-compression wrapper (§3.12), whose decompressed bytes form the same plain body.
• Terminator. A single 0x00 byte (a zero-length row group) closes the stream.

2²⁰ (1,048,576) rows per group is the canonical chunk size — cx canonical and cx hash re-chunk a CXCol input to it, and a writer that wants cross-system hash invariance uses it. A different chunk size produces a valid CXCol file with a different SHA-256.

Content-addressable identity

A CXCol file's SHA-256 is computed over its canonical uncompressed bytes — the bytes you would see if every 0x90 page-compression wrapper were unwrapped to its 0x01 plain form. Compression is a transport and storage choice; identity is a content choice; the format keeps them orthogonal, and the data_bin_compression cases pin that a compressed and a plain encoding of one table share one address.

• Same data + same chunking + same schema → same hash, whatever the zstd level, and whether compression was applied at all.
• Different chunk size → different hash. Chunk boundaries are part of the canonical form; the 2²⁰-rows-per-group convention is the cross-writer rule.
• The hash composes with hash. The same SHA-256 primitive that addresses a CX document addresses a CXCol file — caching, dedup, signing and content-addressed storage treat both the same.

Streaming reader / writer

The streaming surface is the C ABI reader / writer family in spec/03-approved/core/abi.md §2.10 (capability bit 21): cx_table_reader_open / cx_table_reader_open_fd / cx_table_reader_schema / cx_table_reader_next / cx_table_reader_close, and cx_table_writer_open / cx_table_writer_open_fd / cx_table_writer_emit_row_group / cx_table_writer_close_get_bytes / cx_table_writer_close. A reader pulls one row group at a time; a writer pushes one at a time and closes to the framed bytes. Each binding wraps the family in its host's idiom — a context-managed reader and writer where the host has them, the raw symbols where it does not — and bindings says which binding ships which.

The memory budget is one row group in flight per stage. A reader chained into a CXPath filter chained into a Parquet writer holds three row groups at once, whatever the table's logical row count. See streaming for the directive-level view.

Schema-driven mode

When both sides of the wire hold the schema, header flag bit 1 enables schema-driven mode: per-value type tags are omitted for the values the schema declares, and reader and writer walk the schema in lockstep with the data. Wire density approaches Parquet's typed encoding; the data_bin_schema_driven cases pin the bytes, and a closed-mode schema refuses a column it did not declare.

Schema-driven mode is opt-in — cx_to_data_bin --schema-driven at the ABI, the corresponding call in each binding — and a schema in scope never silently promotes the encoder. The schema reference forms (0x10 content-hash, 0x11 inline, 0x12 content-hash + name hint, 0x13 self-describing multihash) are spec/03-approved/core/data-bin.md §3.13.1; the cx table subcommand resolves --schema-from= to a content hash before it emits.

Schema-driven mode needs the reader to retrieve the schema by its content hash. Local lookup is the default; the callback-shaped content-store ABI is a planned follow-up.

Zero-copy handoff

The C Data Interface uses two structs: ArrowSchema (column types and names) and ArrowArray (per-batch buffers — null bitmap, offsets, values). CX points these at its own column buffers; an Arrow-native consumer reads from the same memory pages CX wrote.

Ownership crosses the boundary explicitly. The exporter fills an ArrowArrayStream with a release callback; the consumer calls release when it is done; CX's callback frees the underlying CXCol buffers. There is no garbage-collector interplay — a binding holds the stream as an opaque handle until it is finished with it.

Type mapping

The CX ↔ Arrow type map is normative (spec/03-approved/misc/type-mapping.md). In summary:

• int / i64 ↔ Int64; i32 ↔ Int32; the sized variants (i8–u64) map to the corresponding Arrow sized integer types.
• float / f64 ↔ Float64; f32 ↔ Float32; f16 ↔ Float16.
• bool ↔ Boolean (bit-packed in Arrow).
• string ↔ Utf8 (or LargeUtf8 for a column whose total byte length exceeds 2 GB).
• bytes ↔ Binary (or LargeBinary similarly).
• date ↔ Date32 (days since 1970-01-01, proleptic Gregorian).
• datetime ↔ Timestamp(Nanoseconds, UTC) (Arrow format string tsn:UTC).
• decimal ↔ Decimal128 / Decimal256 by coefficient width.
• A nullable column → Arrow's nullable variant (validity bitmap).

Column-type coverage in libcx_arrow is all ten column types — int / i8 / i16 / i32 / float / bool / string / date / bytes / datetime (the last as timestamp[ns, UTC]). Deferred: decimal and dictionary-encoded columns — each answers a clear not yet supported error rather than a lossy fallback.

Arrow IPC stream format

Beyond in-process handoff, CXCol emits Arrow IPC streams for cross-process or cross-host transport. The IPC stream is Arrow's own serialization (schema header + record batches + EOF), distinct from the C Data Interface (in-memory structs). The cx table dump FILE --to=arrow [--output=FILE] subcommand (spec/03-approved/misc/cli.md) writes it, and each binding's Arrow module reads and writes it as bytes or as a file.

Arrow Flight (planned)

Arrow Flight is Arrow's gRPC-based RPC framing for cross-process record batches — the natural next layer above the IPC stream: a Flight server publishes batches by descriptor; a client subscribes. CX integration is planned: no Flight surface ships, and the tooling-layer story will be specified when it merges.

For cross-process CXCol transport today, carry Arrow IPC over your existing transport (HTTP, a pipe, a socket); the [?http-client] directive carries an IPC stream as a binary body.

Writing Parquet

The adapter walks framed CXCol bytes → Arrow record batches → the host's Parquet writer; the writer is the host's choice, and the CXCol side provides the Arrow stream. Defaults are the Parquet ecosystem's (Snappy, not zstd), because the output is for Parquet readers that assume Snappy; zstd is CXCol's own default for its 0x90 page compression. Match the row-group size to CXCol's canonical chunk (2²⁰ rows) and the two formats chunk alike.

Reading Parquet

The inverse: the host's Parquet reader produces Arrow record batches, and the bridge imports them as framed CXCol chunked-table bytes — ready for a whole-value read or a streaming reader pass. Column projection and predicate pushdown are the host Parquet reader's work, done before the batches reach CX: a reader given per-row-group statistics skips the groups a predicate rules out, so the I/O saved is proportional to the groups skipped; a file without statistics still reads correctly as a full scan.

CX ↔ Parquet type mapping

The mapping is normative. Most types round-trip cleanly through Arrow; a few CX types have lossy edges, and the bridge reports them loudly rather than coercing silently.

• int / i64 → INT64.
• i32 → INT32; i16 / i8 → INT32 with INT(N, true) logical type; unsigned similarly.
• f64 / float → DOUBLE; f32 → FLOAT; f16 → FIXED_LEN_BYTE_ARRAY(2) / FLOAT16 (the latter non-standard before Parquet 1.13).
• bool → BOOLEAN.
• string → BYTE_ARRAY with the STRING logical type (UTF-8).
• bytes → BYTE_ARRAY.
• date → INT32 with the DATE logical type.
• datetime (UTC) → INT64 with TIMESTAMP(NANOS, isAdjustedToUTC=true).
• decimal → DECIMAL(p, s) (Parquet caps precision at 256 bits).
• A nullable column → OPTIONAL repetition.

A type outside the bridge's coverage (see arrow-type-mapping) refuses with a clear error at the Arrow boundary rather than degrading — the same loud-never-lossy posture as the wire form itself.

Round-trip identity

CX → Parquet → CX is not in general byte-identical. Parquet is a presentation-layer format: footer offsets shift, row-group sizes are writer-dependent, encoding choices (dictionary thresholds, page sizes, the codec) are per writer; many byte-level Parquet files encode the same logical data. CXCol fills exactly that niche — the hashable origin; Parquet is the bridge.

Logical round trip is normative: a CX table emitted as Parquet and re-imported produces the same [table[…]] block with the same column names, types, row count and cell values (modulo the documented lossy edges of 5.5.3) — which is the byte-exact CX round trip the table suite grades, taken through one more format.

Schema identity across the bridge

Parquet output carries the data, not the CX schema reference. Schema identity lives at the CXCol layer: a schema-driven file names its schema by content hash (the §3.13.1 reference forms), and cx table resolves --schema-from= to a content hash before it emits (spec/03-approved/core/data-bin.md §3.13.1). A Parquet round trip re-enters CX as a self-describing table; re-attach the schema by validating, or by re-encoding schema-driven against the same schema hash.

Streaming events

The event vocabulary (spec/03-approved/core/streaming.md) extends the standard StartElement / Scalar / EndElement events with three table-specific variants.

• StartTable — fields: name (string), col_spec (binary blob; the column schema in ast_bin form). Emitted in place of the StartElement that would otherwise wrap a [table[…]] body.
• RowGroup — fields: row_count (u32), payload (binary blob; one row group's column buffers). One event per row group, in source order.
• EndTable — fields: name (string). Closes the sequence in place of the corresponding EndElement.

A non-chunked 0x60 (whole-table) input emits the existing StartElement / per-cell-Scalar / EndElement sequence unchanged; only a 0x63 chunked input produces the new events. A consumer that does not care flattens each RowGroup into StartElement / row / EndElement triples with a helper.

Performance posture

The gate protocol is fixed by spec/03-approved/core/code.md §11.4.4: every measured streaming shape must sustain a mean ≥ 200 MB/s with no single trial below 80 % of the mean, and must demonstrably stream (more than one output chunk, the implementation's own streaming predicate reporting true). Re-run with make bench-code-streaming; the register row is the verdict.

On the write side, the ABI streaming writer compresses a row group only when its body exceeds 64 KiB and compression beats 0.85 of the original size (spec/03-approved/core/abi.md §2.10).

DuckDB

DuckDB consumes an Arrow table registered on a connection — zero-copy from CXCol's column buffers into DuckDB's vectorized executor through the C Data Interface. Export the framed CXCol bytes, register the Arrow table, run SQL; DuckDB's executor reads CX's column buffers during the scan, the result set is materialized in DuckDB's own memory, and the bridge's import brings a result back as framed CXCol bytes — the symmetric inverse.

Polars

Polars is Arrow-native at the buffer level: CX → Polars goes through the same export call — zero-copy for the underlying buffers — and a Polars query runs on the frame from there. In the Rust binding the same handoff runs over the C ABI: the cx_arrow_export_open stream feeds the arrow crate's ffi_stream reader, and Polars consumes the record batches; a packaged convenience wrapper there has not shipped.

pandas

pandas is not zero-copy at the buffer level — its column model is NumPy arrays with type-system mismatches against Arrow (no nullable integers, no first-class strings without StringDtype). CX → pandas is one Arrow round trip plus one pandas conversion: expect one full data copy at the boundary. spec/03-approved/misc/table-api.md §3 declares a to_pandas() adapter slot on the table type that no binding has implemented yet; the explicit Arrow route is the shipped path.

Where zero copy matters, prefer Polars over pandas. pandas stays supported because the data-science ecosystem assumes it, but the cross-boundary cost is real on large tables.

DataFusion (planned)

Apache DataFusion is a Rust query engine over Arrow. Its TableProvider trait accepts arbitrary Arrow record-batch sources, so CXCol is a natural fit: register a .cxbin file as a DataFusion table and run SQL or DataFrame queries with the same zero-copy guarantees as Polars.

Status: planned — no packaged CX-side convenience layer ships. An adopter who needs DataFusion today wires it by hand: cx_arrow_export_open over the C ABI into the arrow crate's ffi_stream reader, and the record batches into a DataFusion MemTable. The path works; it lacks the convenience layer.

Spark and cluster compute

Spark reads and writes Parquet natively, so CXCol's Parquet bridge (see parquet-bridge) is the integration path: emit .parquet from CX, point Spark at the directory, run Spark SQL. For zero copy in cluster contexts, Arrow Flight is the long-term path; status is planned per arrow-flight.

The Parquet bridge is one-way at this layer — Spark's writes come back through cx table load output.parquet --to=cx. For a round-trip workflow, keep the CXCol file as the canonical origin and treat Spark's outputs as derived artifacts.