#colenc

• Design, milestones and acceptance criteria: docs/ROADMAP.md
• Next work packages and exit criteria: docs/NEXT-STEPS.md
• Sketch contracts and their acceptance criteria: docs/SKETCHES.md
• Acceptance rig: lives in the source repository under harness/ (not shipped in the package); the summary is below
• Throughput scoreboard: docs/scoreboard.md
• License: Apache-2.0

#Status

#Quick start

moon run examples/encode_demo
moon run examples/encode_demo | sort -t$'\t' -k4,4n

#Build and test

moon check --deny-warn        # type-check, warnings are errors
moon test                     # wasm-gc, the default target
moon test --target native     # and js: backend-differing paths need both
moon package --list           # what a release archive would contain

#API

let column = @colenc_types.ColBuf::Int64s([1L, 4L, 9L])
let out = Buffer()
let stats = @colenc.encode_column(
  column, @colenc_types.DeltaBinaryPacked, out,
) catch {
  e => abort(@colenc_types.EncodingError::message(e))
}
let back = @colenc.decode_column(
  out.to_bytes(), @colenc_types.Int64, @colenc_types.DeltaBinaryPacked, 3,
)

let left = @colenc_encoders.RoaringBitmap::from_ids([1, 2, 3, 100000])
let right = @colenc_encoders.RoaringBitmap::from_ids([3, 4, 100000])
let both = left.intersect(right)          // {3, 100000}
let bytes = left.to_bytes()               // portable Roaring
let back = @colenc_encoders.RoaringBitmap::from_bytes(bytes)

#Encodings

#Scoreboard

#Boundaries (v1)

• No byte-stream compression (DEFLATE / ZSTD / Snappy / Brotli / LZ4): a different layer, composable after column encoding
• Flat columns only: definition/repetition levels are the caller's job
• No public container API: the acceptance rig has a minimal Parquet writer, but it is a judge target, not a promise
• Dictionary fallback is per column (returns None), not parquet-cpp's mid-column fallback
• FSST framing is colenc's own; Parquet has no native FSST, so no byte diff or container check applies, and the framing falls back to stored raw values when the table does not pay for itself
• The Roaring bitmap is a set encoding: a column is the set of its ids, so order and duplicates do not survive a round trip and a decode returns them ascending. Ids are non-negative INT32 (0..2^31-1); the format's upper u32 half needs a column type this model does not have yet and is rejected rather than truncated. Run containers are written only when run_optimize is asked for, matching the reference's incremental path, and decoding accepts them either way. It is a bitmap index, not a Parquet page encoding, so the container interop layer does not apply
• Set operations return bitmaps that own their containers: mutating a union or intersect result never rewrites an operand, and mutating an operand never rewrites an existing result
• An add after run_optimize de-optimises the run container into an array or bitmap, so that sequence is not byte-identical to CRoaring (which keeps the run); the criteria there are round trip plus the container invariants, not the byte diff. Call run_optimize again after a batch of adds to re-adopt runs
• No SIMD kernels: scalar + lookup tables first

#Neighbours and the seams

#Acceptance

#Layout

types/        neutral model: PhysicalType / Encoding / ColBuf / errors
primitives/   L1: varint, zigzag, bit packing, bit width
encoders/     L2: PLAIN, RLE hybrid, dictionary, delta family, BSS, FSST, Roaring
sketches/     approximate aggregations: HyperLogLog and t-digest
facade.mbt    L3: encode_column / decode_column + dictionary entry points
examples/     runnable demo
harness/      acceptance rig: fixtures, byte diffs, containers, judges
docs/         public roadmap, and the throughput scoreboard

#License