sketch

    MoonBit probabilistic data structures library (HyperLogLog, Count-Min Sketch, Cuckoo Filter)

    probabilistic
    hyperloglog
    count-min-sketch
    cuckoo-filter
    data-structures
    streaming
    Download zip
    Version
    0.1.0
    License
    Apache-2.0
    Last updated
    6 months ago
    Downloads
    22

    #hiroyannnn/sketch

    MoonBit probabilistic data structures library — HyperLogLog, Count-Min Sketch, and Cuckoo Filter.

    #Installation

    moon add hiroyannnn/sketch

    Import in moon.pkg:

    import { "hiroyannnn/sketch" @sketch, }

    #Usage

    #HyperLogLog — cardinality estimation

    ///|
    test "HyperLogLog basic" {
    let hll = try! @sketch.HyperLogLog::new()
    for i in 0..<1000 {
    hll.add("item-\{i}")
    }
    let est = hll.count()
    // Estimate within ~2 % of 1000
    assert_true(est > 950.0 && est < 1050.0)
    }

    ///|
    test "HyperLogLog merge" {
    let a = try! @sketch.HyperLogLog::new()
    let b = try! @sketch.HyperLogLog::new()
    a.add("x")
    b.add("y")
    let merged = try! a.merge(b)
    assert_true(merged.count() >= 1.0)
    }

    ///|
    test "HyperLogLog invalid precision" {
    inspect(
    try! @sketch.HyperLogLog::new(precision=3),
    content="Err(InvalidPrecision)",
    )
    }

    #Count-Min Sketch — frequency estimation

    ///|
    test "CountMinSketch basic" {
    let cm = try! @sketch.CountMinSketch::new()
    cm.add("apple", count=5)
    cm.add("banana")
    assert_true(cm.estimate("apple") >= 5)
    assert_true(cm.estimate("banana") >= 1)
    }

    ///|
    test "CountMinSketch merge" {
    let cm1 = try! @sketch.CountMinSketch::new()
    let cm2 = try! @sketch.CountMinSketch::new()
    cm1.add("x", count=3)
    cm2.add("x", count=4)
    let merged = try! cm1.merge(cm2)
    assert_true(merged.estimate("x") >= 7)
    }

    #Cuckoo Filter — set membership

    ///|
    test "CuckooFilter basic" {
    let cf = @sketch.CuckooFilter::new()
    assert_true(cf.insert("hello"))
    assert_true(cf.contains("hello"))
    assert_true(!cf.contains("world"))
    assert_true(cf.remove("hello"))
    assert_true(!cf.contains("hello"))
    }

    #API reference

    #SketchError

    VariantMeaning
    InvalidPrecisionHyperLogLog precision outside [4, 18]
    InvalidDimensionwidth, depth, capacity, or bucket_size is not positive
    PrecisionMismatchMerging two sketches with different parameters

    #HyperLogLog

    FunctionSignatureDescription
    HyperLogLog::new(precision? : Int) -> HyperLogLog raise SketchErrorCreate (default precision 14)
    add(HyperLogLog, String) -> UnitAdd a value
    count(HyperLogLog) -> DoubleEstimate cardinality
    merge(HyperLogLog, HyperLogLog) -> HyperLogLog raise SketchErrorMerge two sketches

    #CountMinSketch

    FunctionSignatureDescription
    CountMinSketch::new(width? : Int, depth? : Int) -> CountMinSketch raise SketchErrorCreate (default width 1000, depth 5)
    add(CountMinSketch, String, count? : Int) -> UnitIncrement frequency
    estimate(CountMinSketch, String) -> IntQuery frequency
    merge(CountMinSketch, CountMinSketch) -> CountMinSketch raise SketchErrorMerge two sketches

    #CuckooFilter

    FunctionSignatureDescription
    CuckooFilter::new(capacity? : Int, bucket_size? : Int) -> CuckooFilter raise SketchErrorCreate
    insert(CuckooFilter, String) -> BoolInsert (false if full)
    contains(CuckooFilter, String) -> BoolMembership query
    remove(CuckooFilter, String) -> BoolRemove (false if not found)
    count(CuckooFilter) -> IntNumber of stored items

    #Acknowledgments

    See NOTICE for algorithm citations and license information.

    #License

    Apache-2.0

    CountMinSketch

    A Count-Min Sketch frequency estimator.

    CuckooFilter

    A Cuckoo Filter for approximate set membership testing.

    HyperLogLog

    A HyperLogLog cardinality estimator.

    SketchError

    Error type for sketch operations.

    • PrecisionMismatch — two sketches have incompatible parameters.
    • InvalidPrecision — precision is outside the allowed range (4–18).
    • InvalidDimension — width or depth is not positive.

    Source Files