🥮 mooncakes.io

    moon-distinct-sketch

    MoonBit streaming sketches with bounded windows, metrics, validated frames, and distributed merge planning

    cardinality
    hyperloglog
    sketch
    metrics
    streaming
    Download zip
    Author
    Version
    0.2.1
    License
    Apache-2.0
    Repository
    Last updated
    2 months ago
    Downloads
    34

    No dependencies

    #moon-distinct-sketch

    CI License

    moon-distinct-sketch is a MoonBit library for memory-efficient cardinality, frequency, similarity, quantile, and multi-tenant streaming analytics. It combines probabilistic sketches with bounded windows, validated wire frames, deterministic merge planning, and a small command-line benchmark.

    #Capabilities

    AreaPackages and use cases
    Cardinalityhll, hllplus, linear, and window for distinct users, devices, and time windows
    Frequencycountmin, topk, and bloom for weighted counts, heavy hitters, admission, and duplicate filtering
    Similarityminhash and multitenant for Jaccard-style set comparison and tenant UV
    Distributionstream for event-time aggregation, watermarks, routing, reordering, checkpoints, and snapshot catalogs
    Summariesquantile and metrics for bounded ranks, histograms, registry queries, deltas, and health exports
    Interchangeserde for legacy sketch formats plus checksummed versioned frames and frame batches
    Coordinationdistributed for partition validation, replay protection, merge reports, and reduction traces

    The APIs are bounded and deterministic where ordering is observable. Invalid weights, stale events, incompatible configurations, malformed frames, and duplicate sequences are rejected explicitly.

    #Quick start

    Install the current stable MoonBit toolchain, then fetch dependencies and run the checks:

    moon update
moon check --target all --deny-warn
moon test --target wasm-gc --deny-warn

    A minimal HLL use case:

    let sketch = @hll.HyperLogLog::new(14)
    sketch.add_string("user-42")
    let estimate = sketch.cardinality()
    println(estimate.to_string())

    For a package published from this repository, import the module as yhsrtty/moon-distinct-sketch and use the package aliases declared in each moon.pkg.

    #CLI

    The executable is intentionally deterministic for demos and benchmark comparisons:

    moon run cmd/main -- help
moon run cmd/main -- demo
moon run cmd/main -- version
moon run cmd/main -- benchmark

    benchmark prints CSV with the fields algorithm,input_count,estimate,tracked_items,serialized_bytes,repeat_count,elapsed_ms,throughput_events_per_second. It reports deterministic algorithm output, available codec sizes, and measured suite-level wall-clock throughput; timing values are specific to the runner.

    #Architecture

    event records ──> reorder buffer ──> watermark tracker ──> partition router
                                                            │
                 ┌──────────────────────────────────────────┘
                 ▼
       StreamAggregator per shard
        ├─ MetricRegistry (counter/gauge/histogram/distinct)
        ├─ SpaceSaving heavy hitters
        └─ HLL-backed window snapshots

sketches ──> validated snapshots ──> versioned serde frames ──> distributed merge plans

    The main packages are organized as one MoonBit package per directory. src/utils provides arithmetic, sorting, hashing, and deterministic support contracts; higher-level packages depend only on the lower-level sketches they compose.

    #Benchmark

    The following table is a real run of moon run cmd/main -- benchmark on 2026-08-19 with MoonBit 0.1.20260814 / moonc 0.10.8+8606a5800 on the local Windows development environment. The workload processes 6,000 events across three identical repeats and 500 distinct keys; the values are estimates, not exact-count assertions. elapsed_ms and throughput_events_per_second measure the complete multi-sketch input loop, so they are reported consistently on each algorithm row rather than presented as per-algorithm timings.

    algorithminput_countestimatetracked_itemsserialized_bytesrepeat_countelapsed_msthroughput_events_per_second
    hll600050004010315040000
    hllplus6000497.4717286076932500315040000
    countmin60001200315040000
    space_saving60006000200315040000
    quantile6000464641594315040000
    histogram6000499.55126315040000
    bloom6000020000315040000
    stream6000482.884967649249610315040000

    To reproduce the table, use the CLI command above. The implementation uses fixed seeds and fixed input sizes so that algorithm output can be compared across toolchain targets. serialized_bytes is populated for the codecs currently available in this repository; zero means that an algorithm-specific wire codec is not claimed. Wall-clock values are environment-specific and should be compared only on the same runner class.

    #Tests and quality gates

    The repository currently contains 174 passing tests, including malformed-frame, merge-compatibility, empty-input, zero-size, negative-timestamp, late-event, duplicate-sequence, session-bridge, snapshot-validation, and capacity-boundary cases.

    The auditable source counter reports 8,241 lines of tracked production .mbt implementation and 2,560 lines of tracked test .mbt code across 136 tracked MoonBit files. It excludes _build, generated .mbti interfaces, benchmark output, and duplicated artifacts. Re-run the counter with:

    powershell -NoProfile -ExecutionPolicy Bypass -File tools/count-moonbit-lines.ps1

    Local quality gates:

    moon fmt --check
moon info
moon check --target all --deny-warn
moon test --target wasm-gc --deny-warn
moon coverage analyze
git diff --exit-code

    On Unix, CI also runs the native-capable suite with moon test --target all --deny-warn.

    #CI

    GitHub Actions runs on Ubuntu, macOS, and Windows. It installs MoonBit through the official installer, verifies the toolchain, updates dependencies, checks formatting and generated interface drift, runs all-target no-warning checks, runs native-capable tests on Unix and the portable wasm-gc backend on Windows, executes coverage, and smoke-tests the CLI. Pull requests and pushes to main use the same gates.

    #Versioning and Mooncakes

    The module metadata is in moon.mod, currently version 0.2.1, with the repository namespace yhsrtty/moon-distinct-sketch. Compatible feature additions increment the minor version; fixes increment the patch version.

    After logging in locally with moon login, publish a checked-out release with:

    moon publish --frozen

    The repository also includes a manual publish.yml workflow. It requires a repository secret named MOONCAKES_CREDENTIALS_JSON containing the credentials JSON produced by moon login; the workflow writes it only for the publish job and removes it on exit. Ordinary pushes never publish.

    #License

    This project is available under the Apache License 2.0.

    Powered by MoonBit

    Site sourceReport issueModulesSkillsCLIBuild queueStatistics

    © 2026 mooncakes.io