moonbitstack/moonstl/sketch does not have a README file

    BloomFilter

    pub struct BloomFilter[T] {
    hash_seed : Int
    hashes : Int
    expected_items : Int
    false_positive_rate : Double
    growth : Growth
    layers : Array[BloomLayer]
    items : Int64
    }

    A Bloom filter answers membership with no false negatives: an item it has taken in is always reported as possibly present. Absent items are reported as present at most at the requested rate, and the bit count follows from the expected item count and that rate, so no caller picks a size by hand.

    The item type is hashed with the filter's seed, which is stored with the bits so that a decoded filter answers exactly as the one that wrote it.
    impl Eq for BloomFilter[T]
    impl Hash for BloomFilter[T]
    impl Show for BloomFilter[T]

    BloomFilter::bit_count

    fn[T] BloomFilter::bit_count(self : BloomFilter[T]) -> Int

    The number of bits in the filter, summed over layers.

    BloomFilter::capacity

    fn[T] BloomFilter::capacity(self : BloomFilter[T]) -> Int

    The number of items the filter is sized for, summed over layers.

    BloomFilter::clear

    fn[T] BloomFilter::clear(self : BloomFilter[T]) -> Unit

    Empties the filter, returning a scalable one to a single layer.

    BloomFilter::copy

    fn[T] BloomFilter::copy(self : BloomFilter[T]) -> BloomFilter[T]

    A deep copy: the two filters share no bit array.

    BloomFilter::decode

    fn[T] BloomFilter::decode(view : BytesView) -> BloomFilter[T] raise
    Refused

    BloomFilter::encode

    fn[T] BloomFilter::encode(self : BloomFilter[T]) -> Bytes

    The filter as bytes: magic, version, seed, shape, then each layer's words little-endian. A decoded filter answers exactly as the one that encoded it.

    BloomFilter::equal

    fn[T] BloomFilter::equal(self : BloomFilter[T], other : BloomFilter[T]) -> Bool

    BloomFilter::estimated_false_positive_rate

    fn[T] BloomFilter::estimated_false_positive_rate(self : BloomFilter[T]) -> Double

    The rate the filter would show at its present fill, (1 - e^(-kn/m))^k per layer, combined across layers. It assumes the hash behaves ideally, so it is an estimate of the design, not a measurement of inserted keys.

    BloomFilter::hash

    fn[T] BloomFilter::hash(self : BloomFilter[T]) -> Int

    BloomFilter::hash_combine

    fn[T] BloomFilter::hash_combine(self : BloomFilter[T], hasher : Hasher) -> Unit

    BloomFilter::insert

    fn[T : Hash] BloomFilter::insert(self : BloomFilter[T], item : T) -> Bool

    Adds an item and reports whether the filter changed. true means the item was certainly absent before: at least one of its bits was clear. A false return leaves the filter untouched.

    A Scalable filter opens a new layer first when the newest one is full.

    BloomFilter::intersection

    fn[T] BloomFilter::intersection(self : BloomFilter[T], other : BloomFilter[T]) -> BloomFilter[T] raise
    Refused

    The intersection of two filters of the same shape, by a bitwise and. An item in both is still reported present; an item in only one has at least one clear bit in the other, so it usually drops out.

    BloomFilter::is_empty

    fn[T] BloomFilter::is_empty(self : BloomFilter[T]) -> Bool

    Whether the filter holds no items.

    BloomFilter::length

    fn[T] BloomFilter::length(self : BloomFilter[T]) -> Int64

    The number of items the filter has taken in. An item the filter already reported as present does not increase it, so this is the n of the rate formula.

    BloomFilter::may_contain

    fn[T : Hash] BloomFilter::may_contain(self : BloomFilter[T], item : T) -> Bool

    Whether the item may be present. It is always true for an item that has been inserted, and may be true for an absent one at the filter's rate.

    BloomFilter::new

    fn[T] BloomFilter::new(expected_items~ : Int, bloom_options? : BloomOptions, hash_seed? : Int) -> BloomFilter[T] raise
    Refused

    Builds a filter sized for expected_items at bloom_options.false_positive_rate.

    The bit count is m = -k n / @math.ln(1 - p^(1/k)) with k = log2(1/p), the standard optimum; with the default rate of 1% that is 9.6 bits per item and 7 probes. A Scalable filter starts with one layer and adds more as it fills.

    test {
    let filter : @sketch.BloomFilter[String] = @sketch.BloomFilter::new(
    expected_items=1000,
    ) catch {
    _ => abort("valid parameters")
    }
    assert_false(filter.may_contain("absent"))
    assert_true(filter.insert("present"))
    assert_true(filter.may_contain("present"))
    }

    BloomFilter::not_equal

    fn[T] BloomFilter::not_equal(x : BloomFilter[T], y : BloomFilter[T]) -> Bool

    BloomFilter::output

    fn[T] BloomFilter::output(self : BloomFilter[T], logger : &Logger) -> Unit

    BloomFilter::to_string

    fn[T] BloomFilter::to_string(self : BloomFilter[T]) -> String

    BloomFilter::union

    The union of two filters of the same shape, by a bitwise or. Items that were in either filter are reported present, at a rate no worse than the worse of the two.

    BloomLayer

    pub struct BloomLayer {
    words : Array[UInt64]
    bit_count : Int
    capacity : Int
    items : Int64
    rate : Double
    }

    One bloom of a filter. A Fixed filter holds exactly one; a Scalable filter adds one when the newest is full.

    BloomOptions

    pub(all) struct BloomOptions {
    false_positive_rate : Double
    growth : Growth
    } derive(Eq,
    Debug
    )

    The two knobs of a Bloom filter. A record rather than two parameters, because callers change one of them far more often than both.

    BloomOptions::equal

    BloomOptions::not_equal

    fn BloomOptions::not_equal(x : BloomOptions, y : BloomOptions) -> Bool

    CountMinSketch

    pub struct CountMinSketch[T] {
    seed : Int
    width : Int
    depth : Int
    counters : Array[Int]
    total : Int64
    }

    A sketch that counts: the estimate for an item is the smallest counter its probes land on, which never underestimates because every counter it reads was incremented by that item too. The width and depth come from the error the caller tolerates, so nobody picks them by hand.

    test {
    let sketch : @sketch.CountMinSketch[String] = @sketch.CountMinSketch::new(
    epsilon=0.01,
    confidence=0.99,
    )
    for _ in 0..<5 {
    sketch.insert("a")
    }
    assert_true(sketch.estimate("a") >= 5)
    assert_true(sketch.estimate("b") == 0)
    assert_true(sketch.total() == 5L)
    }

    CountMinSketch::clear

    fn[T] CountMinSketch::clear(self : CountMinSketch[T]) -> Unit

    CountMinSketch::copy

    fn[T] CountMinSketch::copy(self : CountMinSketch[T]) -> CountMinSketch[T]

    CountMinSketch::decode

    fn[T] CountMinSketch::decode(view : BytesView) -> CountMinSketch[T] raise
    Refused

    CountMinSketch::depth

    fn[T] CountMinSketch::depth(self : CountMinSketch[T]) -> Int

    CountMinSketch::encode

    fn[T] CountMinSketch::encode(self : CountMinSketch[T]) -> Bytes

    CountMinSketch::estimate

    fn[T : Hash] CountMinSketch::estimate(self : CountMinSketch[T], item : T) -> Int

    CountMinSketch::insert

    fn[T : Hash] CountMinSketch::insert(self : CountMinSketch[T], item : T) -> Unit

    Counts one occurrence. The conservative update raises only the counters that hold the current minimum, which is what keeps the estimate from drifting up faster than it has to.

    CountMinSketch::insert_by

    fn[T : Hash] CountMinSketch::insert_by(self : CountMinSketch[T], item : T, amount : Int) -> Unit

    Adds amount occurrences at once.

    CountMinSketch::is_empty

    fn[T] CountMinSketch::is_empty(self : CountMinSketch[T]) -> Bool

    CountMinSketch::merge

    Adds another sketch of the same shape into this one.

    CountMinSketch::new

    fn[T] CountMinSketch::new(epsilon? : Double, confidence? : Double, hash_seed? : Int) -> CountMinSketch[T] raise
    Refused

    epsilon bounds the overestimate as a share of the total, confidence is how sure that bound is. The width is ceil(e / epsilon) and the depth ceil(ln(1 / (1 - confidence))), the derivation in Cormode and Muthukrishnan.

    CountMinSketch::total

    fn[T] CountMinSketch::total(self : CountMinSketch[T]) -> Int64

    CountMinSketch::width

    fn[T] CountMinSketch::width(self : CountMinSketch[T]) -> Int

    CuckooFilter

    pub struct CuckooFilter[T] {
    seed : Int
    slots : Array[Int]
    buckets : Int
    bucket_size : Int
    bits : Int
    mask : UInt64
    items : Int64
    }

    A filter that can forget: every item lives as a short fingerprint in one of two buckets, and the second bucket is computed from the fingerprint alone, so a removal does not have to remember where the item went. An insert that cannot place its fingerprint after a bounded number of kicks reports failure instead of looping, which is the one thing a Cuckoo filter trades away: its false-positive rate, unlike a Bloom filter's, stays flat as it fills.

    test {
    let filter : @sketch.CuckooFilter[String] = @sketch.CuckooFilter::new(
    expected_items=100,
    )
    assert_true(filter.insert("a"))
    assert_true(filter.may_contain("a"))
    assert_true(filter.remove("a"))
    assert_true(!filter.may_contain("a"))
    }

    CuckooFilter::capacity

    fn[T] CuckooFilter::capacity(self : CuckooFilter[T]) -> Int

    CuckooFilter::clear

    fn[T] CuckooFilter::clear(self : CuckooFilter[T]) -> Unit

    CuckooFilter::copy

    fn[T] CuckooFilter::copy(self : CuckooFilter[T]) -> CuckooFilter[T]

    CuckooFilter::decode

    fn[T] CuckooFilter::decode(view : BytesView) -> CuckooFilter[T] raise
    Refused

    CuckooFilter::encode

    fn[T] CuckooFilter::encode(self : CuckooFilter[T]) -> Bytes

    CuckooFilter::estimated_false_positive_rate

    fn[T] CuckooFilter::estimated_false_positive_rate(self : CuckooFilter[T]) -> Double

    The share of lookups that a table of this load factor reports wrongly, from the paper's 2 b / 2^f.

    CuckooFilter::insert

    fn[T : Hash] CuckooFilter::insert(self : CuckooFilter[T], item : T) -> Bool

    Stores the item and reports whether it found a slot. false means the table is too full for this fingerprint, not that the item is absent: the filter is unchanged and the caller should rebuild it larger.

    CuckooFilter::is_empty

    fn[T] CuckooFilter::is_empty(self : CuckooFilter[T]) -> Bool

    CuckooFilter::length

    fn[T] CuckooFilter::length(self : CuckooFilter[T]) -> Int64

    CuckooFilter::may_contain

    fn[T : Hash] CuckooFilter::may_contain(self : CuckooFilter[T], item : T) -> Bool

    CuckooFilter::new

    fn[T] CuckooFilter::new(expected_items~ : Int, cuckoo_options? : CuckooOptions, hash_seed? : Int) -> CuckooFilter[T] raise
    Refused

    CuckooFilter::remove

    fn[T : Hash] CuckooFilter::remove(self : CuckooFilter[T], item : T) -> Bool

    Removes one copy of the item and reports whether a fingerprint matched.

    CuckooOptions

    pub(all) struct CuckooOptions {
    fingerprint_bits : Int
    bucket_size : Int
    } derive(Eq,
    Debug
    )

    The shape of a Cuckoo filter. A fingerprint of fingerprint_bits is short enough that the whole table fits in cache and long enough that a false positive stays rare; four slots per bucket is the number the paper measured.

    CuckooOptions::equal

    CuckooOptions::not_equal

    fn CuckooOptions::not_equal(x : CuckooOptions, y : CuckooOptions) -> Bool

    Growth

    pub(all) enum Growth {
    Fixed
    Scalable
    } derive(Eq,
    Debug
    )

    How a Bloom filter behaves once it holds more items than it was sized for.

    Growth::equal

    fn Growth::equal(Growth, Growth) -> Bool

    Growth::not_equal

    fn Growth::not_equal(x : Growth, y : Growth) -> Bool

    Growth::to_repr

    HyperLogLog

    pub struct HyperLogLog[T] {
    seed : Int
    precision : Int
    registers : Array[Int]
    }

    An estimate of how many distinct items were added, from one small register per bucket: each item sets its bucket to the longest run of leading zeroes its hash produced, and the harmonic mean of the registers is the estimate. The error is about 1.04 / sqrt(m), so a thousand registers buy three percent, and neither the memory nor the insert cost depends on the count.

    test {
    let log : @sketch.HyperLogLog[Int32] = @sketch.HyperLogLog::new(precision=12)
    for i in 0..<1000 {
    log.add(i)
    }
    let estimate = log.estimate()
    assert_true(estimate > 900.0)
    assert_true(estimate < 1100.0)
    }

    HyperLogLog::add

    fn[T : Hash] HyperLogLog::add(self : HyperLogLog[T], item : T) -> Unit

    HyperLogLog::clear

    fn[T] HyperLogLog::clear(self : HyperLogLog[T]) -> Unit

    HyperLogLog::copy

    fn[T] HyperLogLog::copy(self : HyperLogLog[T]) -> HyperLogLog[T]

    HyperLogLog::decode

    fn[T] HyperLogLog::decode(view : BytesView) -> HyperLogLog[T] raise
    Refused

    HyperLogLog::encode

    fn[T] HyperLogLog::encode(self : HyperLogLog[T]) -> Bytes

    HyperLogLog::estimate

    fn[T] HyperLogLog::estimate(self : HyperLogLog[T]) -> Double

    HyperLogLog::is_empty

    fn[T] HyperLogLog::is_empty(self : HyperLogLog[T]) -> Bool

    HyperLogLog::merge

    fn[T] HyperLogLog::merge(self : HyperLogLog[T], other : HyperLogLog[T]) -> Unit raise
    Refused

    Folds another log of the same precision into this one, register by register.

    HyperLogLog::new

    fn[T] HyperLogLog::new(precision? : Int, hash_seed? : Int) -> HyperLogLog[T] raise
    Refused

    precision is the number of bits that choose the bucket, so the register count is 2^precision. The usual range is 4 to 18: below 4 the error is unusable, above 18 the register array is no longer small.

    HyperLogLog::precision

    fn[T] HyperLogLog::precision(self : HyperLogLog[T]) -> Int

    TDigest

    pub struct TDigest {
    compression : Double
    means : Array[Double]
    weights : Array[Double]
    buffer : Array[Double]
    count_ : Double
    minimum : Double
    maximum : Double
    }

    A digest of a distribution: the values are grouped into centroids, each a mean with a weight, and the groups are kept small where the quantiles are dense — near the ends — so a quantile costs the same whether the digest has seen a thousand values or a million. The compression parameter is how many groups the digest is willing to keep; a hundred is the usual choice for a relative error of about one percent at the tails.

    test {
    let digest : @sketch.TDigest = @sketch.TDigest::new()
    for i in 0..<1000 {
    digest.add(i.to_double())
    }
    let median = digest.quantile(0.5)
    assert_true(median > 450.0)
    assert_true(median < 550.0)
    assert_true(digest.count() == 1000.0)
    }

    TDigest::add

    fn TDigest::add(self : TDigest, value : Double) -> Unit

    TDigest::add_weighted

    fn TDigest::add_weighted(self : TDigest, value : Double, weight : Double) -> Unit

    TDigest::cdf

    fn TDigest::cdf(self : TDigest, value : Double) -> Double

    The share of the values at or below value.

    TDigest::clear

    fn TDigest::clear(self : TDigest) -> Unit

    TDigest::compress

    fn TDigest::compress(self : TDigest) -> Unit

    Folds the buffered values into the centroids and merges neighbours that have grown past the size the compression allows at their quantile.

    TDigest::copy

    fn TDigest::copy(self : TDigest) -> TDigest

    TDigest::count

    fn TDigest::count(self : TDigest) -> Double

    TDigest::decode

    fn TDigest::decode(view : BytesView) -> TDigest raise
    Refused

    TDigest::encode

    fn TDigest::encode(self : TDigest) -> Bytes

    TDigest::is_empty

    fn TDigest::is_empty(self : TDigest) -> Bool

    TDigest::maximum

    fn TDigest::maximum(self : TDigest) -> Double

    TDigest::merge

    fn TDigest::merge(self : TDigest, other : TDigest) -> Unit

    Adds every value of another digest, keeping the centroids it had.

    TDigest::minimum

    fn TDigest::minimum(self : TDigest) -> Double

    TDigest::new

    fn TDigest::new(compression? : Double) -> TDigest raise
    Refused

    TDigest::quantile

    fn TDigest::quantile(self : TDigest, q : Double) -> Double

    The value at quantile q: interpolated inside the centroid that covers that share of the distribution, and between the extreme and the nearest centroid outside it.

    ThetaSketch

    pub struct ThetaSketch[T] {
    seed : Int
    nominal : Int
    hashes : Array[UInt64]
    theta : UInt64
    }

    A sketch that keeps the k smallest hashes it has seen and reports a count from the largest one it kept: if the threshold sits at theta, then k / theta scales a sample of k hashes to the whole set. Because the kept hashes are exactly the ones below the threshold, the estimate is a maximum-likelihood one, and two sketches combine by keeping the union's k smallest.

    test {
    let sketch : @sketch.ThetaSketch[Int32] = @sketch.ThetaSketch::new(
    nominal_entries=1024,
    )
    for i in 0..<500 {
    sketch.add(i)
    }
    assert_true(sketch.estimate() > 450.0)
    assert_true(sketch.estimate() < 550.0)
    }

    ThetaSketch::a_not_b

    fn[T] ThetaSketch::a_not_b(self : ThetaSketch[T], other : ThetaSketch[T]) -> ThetaSketch[T]

    The hashes of this sketch that the other one does not have.

    ThetaSketch::add

    fn[T : Hash] ThetaSketch::add(self : ThetaSketch[T], item : T) -> Unit

    ThetaSketch::copy

    fn[T] ThetaSketch::copy(self : ThetaSketch[T]) -> ThetaSketch[T]

    ThetaSketch::decode

    fn[T] ThetaSketch::decode(view : BytesView) -> ThetaSketch[T] raise
    Refused

    ThetaSketch::encode

    fn[T] ThetaSketch::encode(self : ThetaSketch[T]) -> Bytes

    ThetaSketch::estimate

    fn[T] ThetaSketch::estimate(self : ThetaSketch[T]) -> Double

    The number of distinct items, exact while the sketch holds fewer than the nominal count and estimated from the threshold after that.

    ThetaSketch::intersection

    fn[T] ThetaSketch::intersection(self : ThetaSketch[T], other : ThetaSketch[T]) -> ThetaSketch[T]

    The hashes both sketches kept, which estimates the intersection of the two sets while both were sampling at the same threshold.

    ThetaSketch::is_empty

    fn[T] ThetaSketch::is_empty(self : ThetaSketch[T]) -> Bool

    ThetaSketch::new

    fn[T] ThetaSketch::new(nominal_entries? : Int, hash_seed? : Int) -> ThetaSketch[T] raise
    Refused

    ThetaSketch::nominal_entries

    fn[T] ThetaSketch::nominal_entries(self : ThetaSketch[T]) -> Int

    ThetaSketch::theta

    fn[T] ThetaSketch::theta(self : ThetaSketch[T]) -> UInt64

    ThetaSketch::union

    fn[T] ThetaSketch::union(self : ThetaSketch[T], other : ThetaSketch[T]) -> ThetaSketch[T]

    Keeps the union's k smallest hashes, which is what makes two sketches combinable without losing the sample.

    bloom_options

    let bloom_options : BloomOptions

    cuckoo_options

    let cuckoo_options : CuckooOptions

    The preset: 12-bit fingerprints and four slots per bucket, as the original paper's default.