moonbit-roaring

    Interoperability-focused RoaringBitmap with automatic run compression and portable serialization

    roaring
    bitmap
    compression
    serialization
    data-structures
    Download zip
    Version
    0.1.0
    License
    Apache-2.0
    Last updated
    last month
    Downloads
    30

    No dependencies

    #moonbit-roaring

    CI

    An interoperability-focused RoaringBitmap implementation for MoonBit. It combines automatic Array/Bitmap/Run representation selection with the Roaring portable binary format.

    The project complements, rather than replaces, kesmeey/RoaringBitmap. That package established the core bitmap and set-operation API in MoonBit; this package focuses on portable-format interoperability, automatic run compression on the normal mutation path, strict decoding, and current-toolchain reproducibility.

    #Why another package?

    The overlap is deliberate and documented: both packages provide 32-bit integer sets and standard set operations. The independently useful additions here are:

    • Portable interoperability — reads and writes the format specified by RoaringFormatSpec, so MoonBit can participate in existing cross-language Roaring data pipelines.
    • Run optimization on normal operations — add, remove, range methods, and all set operations reselect Array, Bitmap, or Run representation by encoded byte cost.
    • Defensive decoding — rejects truncated input, invalid offsets, unordered keys or arrays, overlapping runs, cardinality mismatches, and trailing bytes.
    • Additional bitmap-level APIs — half-open range construction/mutation, lazy iteration, rank, select, multi-way operations, plus compatibility helpers clear, inclusive range, and jaccard_similarity.

    A source-linked, fair comparison is in docs/competition/competitor-research.md.

    #Installation

    moon add xcc-ordinary/moonbit-roaring

    Until the Mooncakes package is published, depend on the Git repository or clone this repository to run the examples and tests.

    #Example

    let sparse = @roaring.RoaringBitmap::from_array([1U, 100U, 1000U])
    let consecutive = @roaring.RoaringBitmap::from_range(0U, 10000U)
    let result = sparse.union(consecutive)

    // The only exposed wire format is the interoperable portable format.
    let bytes = result.serialize()
    let restored = match @roaring.RoaringBitmap::deserialize(bytes) {
    Ok(bitmap) => bitmap
    Err(error) => panic("invalid bitmap: \{error}")
    }

    let first_three = restored.iter().take(3).to_array()
    let count_at_or_below_500 = restored.rank(500U)
    let fifth = restored.select(4)

    Runnable user paths:

    moon run examples/inverted_index moon run examples/portable_window moon run examples/analytics

    They demonstrate postings-list queries, a portable time-window handoff, and bitmap-level cumulative/order analytics respectively.

    #Implemented surface

    • Array, Bitmap, and Run containers
    • standard set operations and multi-way union/intersection
    • half-open range construction, mutation, and containment
    • lazy Iter[UInt], bitmap-level rank and select
    • Roaring portable serialization and strict deserialization
    • immutable compatibility helpers: clear, inclusive range, Jaccard similarity
    • statistics exposing container choices and encoded payload sizes

    #Verification evidence

    • Tests run on wasm, wasm-gc, and js in CI.
    • Official Java-generated files from RoaringFormatSpec/testdata are decoded, checked for known values, and re-encoded byte-for-byte.
    • Six additional fixture shapes are generated by roaring-wasm v1.1.0, a third-party CRoaring-based WASM package. CI verifies deterministic regeneration.
    • Malformed-format tests cover structural and semantic validation, including every truncated prefix of representative serialized data.
    • Release-mode benchmark fixtures and a reproducible baseline are documented in docs/benchmarks.md. They are not a competitor-speed claim.

    moon info moon fmt --check moon check -d moon test moon test --target wasm-gc moon test --target js moon bench --release --target wasm

    #Comparison scope

    As reproduced on 2026-09-17 at competitor commit f19c4977512aa120cd1add61e32f9dec36bd3102:

    Areakesmeey/RoaringBitmapmoonbit-roaring
    Basic 32-bit set operationsYesYes
    Array/Bitmap/Run definitionsYesYes
    Run selection on ordinary mutation pathsOptimizer has no call site at that commitIntegrated into mutations and set operations
    Roaring portable bytesNot found at that commitTested with official testdata
    Range APIInclusive filtering of existing valuesSame compatibility API plus half-open construction/mutation
    IterationCallback traversalLazy Iter[UInt]
    RankContainer-level helperBitmap-level rank and select
    Current toolchainPinned commit has old-generic-syntax parse errors on moon 0.1.20260904Three-target CI

    These are commit- and toolchain-specific observations, not a judgment about the other project's overall quality. Reproduction commands and source links are in the comparison document.

    #Scope and limitations

    • 32-bit unsigned values only; no Roaring64.
    • Portable format only; no native/frozen/mmap formats.
    • Immutable operations; no thread-safe mutable bitmap API.
    • No SIMD-specific kernels and no claim of outperforming other implementations.
    • The native target is not currently part of the verified matrix.

    #License

    Apache-2.0.

    This project is being prepared for the September 2026 MoonBit Hackathon. The resubmission evidence is tracked in docs/competition/resubmission-strategy.zh-CN.md.

    HighContainer

    type HighContainer derive(Eq)

    High-level container: associates a 16-bit key with a container RoaringBitmap partitions 32-bit integers into:
    • high 16 bits → key (bucket index)
    • low 16 bits → stored in container

    RoaringBitmap

    pub struct RoaringBitmap {
    containers : Array[HighContainer]
    size : Int
    } derive(Eq)

    The main RoaringBitmap structure

    RoaringBitmap::add

    fn RoaringBitmap::add(self : RoaringBitmap, value : UInt) -> RoaringBitmap

    Add a single element. Decodes the target bucket's container, inserts the value, then re-optimizes container representation (may upgrade/downgrade between Array/Bitmap/Run).

    RoaringBitmap::add_many

    fn RoaringBitmap::add_many(self : RoaringBitmap, values : Array[UInt]) -> RoaringBitmap

    Add multiple elements. Groups by high key first so each bucket is re-optimized once instead of once per inserted value.

    RoaringBitmap::add_range

    fn RoaringBitmap::add_range(self : RoaringBitmap, start : UInt, end : UInt) -> RoaringBitmap

    Add every integer in the half-open range [start, end). Buckets fully covered by the range collapse to a single Run container in O(1); this is the fast path that makes bulk-loading a consecutive ID range (a common pattern for time-series data and sequentially-assigned IDs) cheap.

    Note: because end is exclusive, this cannot express a range that extends through UInt's maximum value (4294967295) — end would have to overflow. Use add for that specific boundary value if needed.

    RoaringBitmap::clear

    Return an empty bitmap. This compatibility helper mirrors the established MoonBit RoaringBitmap API while preserving this package's immutable style: the receiver is not modified.

    RoaringBitmap::contains

    fn RoaringBitmap::contains(self : RoaringBitmap, value : UInt) -> Bool

    Check if element exists

    RoaringBitmap::contains_range

    fn RoaringBitmap::contains_range(self : RoaringBitmap, start : UInt, end : UInt) -> Bool

    Whether every integer in the half-open range [start, end) is present. An empty range (start >= end) is trivially contained.

    RoaringBitmap::deserialize

    fn RoaringBitmap::deserialize(data : Bytes) -> Result[RoaringBitmap, RoaringError]

    Deserialize from the CRoaring-compatible portable binary format. Rejects truncated or malformed input with a RoaringError rather than reading out of bounds.

    RoaringBitmap::difference

    fn RoaringBitmap::difference(self : RoaringBitmap, other : RoaringBitmap) -> RoaringBitmap

    Difference: A \ B (elements in self but not in other). Keys unique to self pass through unchanged; keys unique to other are ignored.

    RoaringBitmap::equals

    fn RoaringBitmap::equals(self : RoaringBitmap, other : RoaringBitmap) -> Bool

    Structural equality by content, not by internal container layout: two bitmaps with the same elements are equal even if one stores a range as a Run and the other happens to store it as an Array.

    RoaringBitmap::from_array

    fn RoaringBitmap::from_array(values : Array[UInt]) -> RoaringBitmap

    Create from an array of values

    RoaringBitmap::from_range

    fn RoaringBitmap::from_range(start : UInt, end : UInt) -> RoaringBitmap

    Create a bitmap containing every integer in the half-open range [start, end).

    RoaringBitmap::get_stats

    fn RoaringBitmap::get_stats(self : RoaringBitmap) -> RoaringStats

    Get detailed statistics about internal structure Useful for verifying container optimization

    RoaringBitmap::intersect

    fn RoaringBitmap::intersect(self : RoaringBitmap, other : RoaringBitmap) -> RoaringBitmap

    Intersection: A ∩ B. Keys present in only one side are dropped without inspecting their container; matching keys are intersected and any container that becomes empty is removed from the result entirely.

    RoaringBitmap::intersect_all

    fn RoaringBitmap::intersect_all(bitmaps : Array[RoaringBitmap]) -> RoaringBitmap

    Intersection of every bitmap in bitmaps. An empty input has no well-defined intersection mathematically; by convention this returns an empty bitmap rather than raising an error.

    RoaringBitmap::is_empty

    fn RoaringBitmap::is_empty(self : RoaringBitmap) -> Bool

    Check if empty

    RoaringBitmap::is_subset

    fn RoaringBitmap::is_subset(self : RoaringBitmap, other : RoaringBitmap) -> Bool

    Check if self is a subset of other

    RoaringBitmap::iter

    fn RoaringBitmap::iter(self : RoaringBitmap) -> Iter[UInt]

    An iterator over every element in ascending order. Decodes one bucket's container at a time rather than materializing the whole bitmap up front (as to_array() does), so bm.iter().take(n) or bm.iter().find_first(f) can stop without paying to decode buckets past where iteration stopped.

    RoaringBitmap::jaccard_similarity

    fn RoaringBitmap::jaccard_similarity(self : RoaringBitmap, other : RoaringBitmap) -> Double

    Jaccard similarity: |A ∩ B| / |A ∪ B|. Two empty sets are considered identical and therefore have similarity 1.0.

    RoaringBitmap::max

    fn RoaringBitmap::max(self : RoaringBitmap) -> UInt?

    Maximum element, if any

    RoaringBitmap::min

    fn RoaringBitmap::min(self : RoaringBitmap) -> UInt?

    Minimum element, if any

    RoaringBitmap::new

    Create an empty RoaringBitmap

    RoaringBitmap::range

    fn RoaringBitmap::range(self : RoaringBitmap, start : UInt, end : UInt) -> RoaringBitmap

    Keep only values in the inclusive interval [start, end].

    This deliberately matches kesmeey/RoaringBitmap::range for migration. For constructing or mutating half-open intervals, prefer from_range, add_range, and remove_range.

    RoaringBitmap::rank

    fn RoaringBitmap::rank(self : RoaringBitmap, value : UInt) -> Int

    Number of elements <= value (CRoaring calls this rank). For example, bm.rank(bm.max().unwrap()) equals bm.size().

    RoaringBitmap::remove

    fn RoaringBitmap::remove(self : RoaringBitmap, value : UInt) -> RoaringBitmap

    Remove a single element

    RoaringBitmap::remove_range

    fn RoaringBitmap::remove_range(self : RoaringBitmap, start : UInt, end : UInt) -> RoaringBitmap

    Remove every integer in the half-open range [start, end). A bucket fully covered by the range is dropped outright rather than decoded.

    RoaringBitmap::select

    fn RoaringBitmap::select(self : RoaringBitmap, index : Int) -> UInt?

    The index-th smallest element (0-indexed), or None if index is out of bounds. Inverse of rank in the sense that bm.select(bm.rank(x) - 1) == Some(x) whenever bm.contains(x).

    RoaringBitmap::serialize

    fn RoaringBitmap::serialize(self : RoaringBitmap) -> Bytes

    Serialize to the CRoaring-compatible portable binary format. This project intentionally exposes only the interoperable format; native and frozen formats are separate future APIs rather than a boolean mode.

    RoaringBitmap::singleton

    fn RoaringBitmap::singleton(value : UInt) -> RoaringBitmap

    Create from a single value

    RoaringBitmap::size

    fn RoaringBitmap::size(self : RoaringBitmap) -> Int

    Get cardinality (number of elements)

    RoaringBitmap::to_array

    fn RoaringBitmap::to_array(self : RoaringBitmap) -> Array[UInt]

    Decode every container back into a flat, ascending Array[UInt]. O(n) in the bitmap's cardinality; prefer contains/size when you don't need every element materialized.

    RoaringBitmap::union

    Union: A ∪ B. Merge-walks both bitmaps' sorted container keys; where a key exists in both, the two containers are combined and re-optimized, so e.g. two small Array containers can come back as a single Run.

    RoaringBitmap::union_all

    fn RoaringBitmap::union_all(bitmaps : Array[RoaringBitmap]) -> RoaringBitmap

    Union of every bitmap in bitmaps, e.g. merging the postings lists of several search terms with a single call instead of folding union manually. Returns an empty bitmap for an empty input.

    RoaringBitmap::xor

    Symmetric difference: A ⊕ B (elements in exactly one of the two sets). Keys present in only one bitmap pass through unchanged; matching keys are xor'd at the container level and dropped if the result is empty.

    RoaringError

    pub enum RoaringError {
    DeserializationError(String)
    } derive(Eq,
    Debug
    )

    Error types

    RoaringStats

    pub struct RoaringStats {
    total_containers : Int
    array_containers : Int
    bitmap_containers : Int
    run_containers : Int
    total_bytes_used : Int
    compression_ratio : Double
    } derive(
    Debug
    )

    Statistics for debugging and optimization verification

    Run

    pub struct Run {
    start : UInt16
    length : UInt16
    } derive(Eq,
    Debug
    )

    A run represents a consecutive range [start, start+length]

    array_container_max_size

    let array_container_max_size : Int

    Container size thresholds

    bitmap_container_size_bytes

    let bitmap_container_size_bytes : Int

    run_container_min_run_length

    let run_container_min_run_length : Int