🥮 mooncakes.io
    Sign in

    moonward

    Resilience without I/O: circuit breaking, rate limiting, adaptive load shedding, backoff, rolling windows and deadlines, with the clock and the randomness passed in.

    resilience
    circuit-breaker
    rate-limit
    load-shedding
    backoff
    retry
    moonbit
    Download zip
    Version
    0.2.0
    License
    Apache-2.0
    Repository
    Last updated
    3 hours ago
    Downloads
    57

    Dependencies

    #moonward

    Resilience for MoonBit without I/O: a circuit breaker, adaptive load shedding, rate limits, quotas, concurrency limits, backoff and deadlines. Every decision is a function of what the caller passes in, so every sequence is reproducible in a test.

    let now = 0L // the caller's clock, in nanoseconds
    let breaker = @moonward.Breaker::new()
    match breaker.allow(now, 0.42) {
      Some(call) => call.pass(now + 30_000_000L) // or call.fail(...)
      None => println("backend refused")
    }
    let shedder = @moonward.Shedder::new()
    match shedder.allow(now, 400) {
      Some(ticket) => ticket.pass(now + 30_000_000L) // or ticket.fail()
      None => println("overloaded")
    }
    let daily = @moonward.Period::new(1000, @moondate.Span::new(days=1L), align=true)
    let quota = daily.take(now) // Under
    let rate = @moonward.Bucket::new(100.0, burst=200.0).take(now) // true
    let wait = @moonward.backoff.delay(2, random=0.5) // 100 ms

    moon run src/examples/tour puts a failing backend behind a breaker, overloads a server behind a shedder and steps once through each limiter.

    #No clock, no dice

    Nothing here reads a clock or draws a random number. now is whatever the caller's clock read, in nanoseconds; where a decision is a roll, the caller passes a draw from [0, 1); where it depends on CPU, the caller passes the reading in thousandths. Durations are @moondate.Span.

    A breaker that read the clock itself could only be tested by waiting, and one that drew its own numbers could not be tested at all. Passing both in turns an hour of traffic into a loop over now.

    #The pieces

    What it answers
    BreakerWhether to call a backend that has been failing, as a probability rather than a switch
    ShedderWhether this server should take on another call
    BucketWhether a call is within a rate, and how long until one would be
    SlidingWhether it is within a quota counted over any window-long stretch
    PeriodThe same quota counted per fixed period, cheaper and coarser
    GateWhether there is room right now, which has nothing to do with time
    BackoffHow long to wait before trying again, and whether to
    DeadlineHow much of a time budget is left
    Window[B]The rolling window the breaker and the shedder read; any type with a Default can be its bucket

    #The breaker

    There is no open, half-open and closed. The breaker keeps ten seconds of outcomes and refuses each call with a probability that grows with the failures, so a struggling backend keeps receiving what it can still serve instead of being cut off and then flooded again on recovery. With accepts successes out of total calls, failing the run of all-failure buckets at the head of the window and working the run of all-success ones:

    With failing and working at zero this is equation 21-1 of the Google SRE book, Handling Overload. The decay of k, the recovery factor and the probe that lets one call through after a silent second are go-zero's, from core/breaker/googlebreaker.go. A refused call is recorded too, so a breaker that is refusing never reads back as idle.

    #The shedder

    A call is refused only when the server is both hot and full, which is go-zero's core/load/adaptiveshedder.go:

    • hot: the CPU is at or over the threshold, or was within the last second and the shedder has refused since;
    • full: both the moving average and the current count of calls in flight exceed what the server has shown it can carry, the busiest bucket's passes per millisecond times the fastest bucket's mean response time, scaled down as the CPU nears 100% but never under a tenth.

    A CPU spike with nothing queued refuses nothing, and neither does a queue on an idle CPU.

    #Where they run

    moonpool puts a breaker, a bucket, a backoff and a shedder in front of a resource pool as a Ward, so a pool stops connecting to a failing backend and spaces out its reconnections; moondb's connection pool takes the same ward. moonzero runs go-zero's breaker, shedder, limiters and timeout middleware on these parts, with go-zero's millisecond clock on the outside.

    #Configuration

    The breaker and the shedder each take one policy record, with a preset of the same name: Breaker::new(sre=...), Shedder::new(shed=...). A policy is built with Sre::new(k=2.0), or from the preset with { ..sre, k: 2.0 }; the two are the same value. There are no per-call overrides, so there is nothing to rank.

    PresetValuesSource
    srek 1.5 decaying to 1.1, protection 5, probe after 1 s, 40 buckets over 10 sgo-zero googlebreaker.go
    shed50 buckets over 5 s, CPU threshold 900‰, 1 s assumed response time, average weight 0.9, 1 s to cool, never under 0.1 of capacitygo-zero adaptiveshedder.go
    backoff3 attempts, 100 ms doubling to 1 s, a fifth either waygRPC proposal A6 for the schedule and spread, as grpc-go and grpc-java draw it; the AWS SDKs' standard mode for the attempts

    Period resets a day after first use unless align is set, which is go-zero's PeriodLimit default. A Deadline with a budget of zero or less never expires, which is how go-zero and gRPC spell no timeout.

    #Licence

    Apache-2.0. See LICENSE.

    Backoff

    source
    pub(all) struct Backoff {
      base : 
    Span
      factor : Double
      cap : 
    Span
      attempts : Int
      jitter : Jitter
    } derive(Eq, 
    Debug    )

    When to try again, and how long to wait first.

    The preset is [backoff]. Build another with [Backoff::new], or update one in place with { ..backoff, attempts: 5 }.

    Backoff::ceiling

    source
    fn Backoff::ceiling(self : Backoff, retry : Int) -> 
    Span

    The undelayed ceiling before the retry-th retry, the first retry being 1: min(base × factor^(retry-1), cap).

    This is what a jitter-free client sleeps and what every other strategy draws within, so it is public: a caller comparing this library against a policy written elsewhere compares this.

    Backoff::delay

    source
    fn Backoff::delay(self : Backoff, retry : Int, random? : Double, previous? : 
    Span    ) -> 
    Span

    How long to wait before the retry-th retry.

    random is a value in [0, 1) that the caller draws; passing the same one twice gives the same delay twice, which is what makes a retry sequence testable. previous is what the last delay turned out to be, which only Decorrelated reads — it grows from where it landed rather than from the attempt number.

    Backoff::equal

    source
    fn Backoff::equal(Backoff, Backoff) -> Bool

    Backoff::more

    source
    fn Backoff::more(self : Backoff, made : Int) -> Bool

    Whether a call that has made made attempts may make another.

    The first attempt is 1, so a policy of three attempts allows two retries.

    Backoff::new

    source
    fn Backoff::new(base? : 
    Span    , factor? : Double, cap? : 
    Span    , attempts? : Int, jitter? : Jitter) -> Backoff

    A backoff by name, every knob with the preset's value.

    Backoff::not_equal

    source
    fn Backoff::not_equal(x : Backoff, y : Backoff) -> Bool

    Backoff::schedule

    source
    fn Backoff::schedule(self : Backoff, randoms? : ArrayView[Double]) -> Array[
    Span    ]

    Every delay a run of retries would wait, for a caller that wants the whole sequence rather than one step — a test, or a log line explaining a plan.

    randoms supplies the draws; when it runs short the remaining steps use zero, which is the low end of whatever strategy is in force.

    Backoff::to_repr

    source

    Breaker

    source
    pub struct Breaker {
      sre : Sre
      win : Window[Tally]
      last : Int64?
    }

    A circuit breaker that sheds a fraction of calls rather than all of them.

    There is no open, half-open and closed: the breaker keeps a rolling window of outcomes and refuses each new call with a probability that grows with the failures in it. A struggling backend keeps receiving as much as it can still serve, instead of being cut off and then flooded again when it recovers.

    With accepts successes out of total calls in the window, failing the run of all-failure buckets at its head and working the run of all-success ones:

    w     = max(k - (k - floor) × failing / size, floor)
drop  = (total - protection - w × accepts) / (total + 1)
drop ×= (size - working) / size

    A drop at or below zero lets everything through, which is the whole healthy case.

    Breaker::allow

    source
    fn Breaker::allow(self : Breaker, now : Int64, roll : Double) -> Promise?

    Ask to make one call at now.

    roll is a draw from [0, 1); the call is refused when it falls under the drop fraction. Some means go ahead, and the caller settles the promise once the call has finished. A refusal is recorded as a Drop, so a breaker that is refusing never reads back as idle and throttles itself off.

    Breaker::new

    source
    fn Breaker::new(sre? : Sre, now? : Int64) -> Breaker

    A breaker with an empty window opening at now.

    Breaker::ratio

    source
    fn Breaker::ratio(self : Breaker, now : Int64) -> Double

    The fraction of calls the breaker would refuse at now, zero while the window is healthy.

    The probe is left out: this is the standing throttle, which is what a dashboard or an alert wants.

    Bucket

    source
    pub struct Bucket {
      rate : Double
      burst : Double
      tokens : Double
      at : Int64
    }

    A token bucket: rate tokens accrue per second up to burst, and each request takes one.

    It is the rate limiter that allows a burst after a quiet period, which is what a limit on a human-facing API wants — nobody should be refused for making two requests in the same second after an hour of making none.

    Time is nanoseconds from the caller's clock, as everywhere in this package.

    Bucket::new

    source
    fn Bucket::new(rate : Double, burst? : Double, now? : Int64) -> Bucket

    A bucket that starts full, which is what lets the first burst through.

    rate is tokens per second and burst is the most it will ever hold. A burst smaller than one means nothing ever passes, so it is raised to one.

    Bucket::next

    source
    fn Bucket::next(self : Bucket, now : Int64, n? : Double) -> 
    Span

    How long until n tokens would be available, without taking them.

    Zero when they are available now. This is what a caller sleeps for rather than spinning, and what a Retry-After header is computed from.

    Bucket::take

    source
    fn Bucket::take(self : Bucket, now : Int64, n? : Double) -> Bool

    Whether n tokens are available now, taking them if they are.

    Bucket::tokens

    source
    fn Bucket::tokens(self : Bucket, now : Int64) -> Double

    How many tokens are in the bucket, for a caller that reports its own headroom.

    Deadline

    source
    pub struct Deadline {
      start : Int64
      budget : Int64
    }

    A time budget measured from the instant it started.

    A budget of zero or less never expires, which is how go-zero, gRPC and most servers spell "no timeout": a configuration that leaves the timeout unset should not time everything out at once.

    Deadline::expired

    source
    fn Deadline::expired(self : Deadline, now : Int64) -> Bool

    Whether the budget is spent at now.

    Deadline::remaining

    source
    fn Deadline::remaining(self : Deadline, now : Int64) -> 
    Span    ?

    What is left of the budget at now, never negative. None when there is no budget, which has no finite remainder.

    Deadline::start

    source
    fn Deadline::start(budget : 
    Span    , now : Int64) -> Deadline

    A deadline of budget starting at now.

    Gate

    source
    pub struct Gate {
      limit : Int
      held : Int
    }

    A count with a ceiling: the concurrency limit a server puts on itself.

    Unlike a rate, this does not depend on time at all — it is how many things are happening at once. A server past its limit answers rather than queues, which is what keeps a queue from becoming the outage.

    Gate::enter

    source
    fn Gate::enter(self : Gate) -> Bool

    Whether there is room, taking it if there is.

    Gate::held

    source
    fn Gate::held(self : Gate) -> Int

    How many are inside.

    Gate::leave

    source
    fn Gate::leave(self : Gate) -> Unit

    Give the room back.

    Gate::new

    source
    fn Gate::new(limit : Int) -> Gate

    A gate that nothing has entered.

    Jitter

    source
    pub(all) enum Jitter {
      Rigid
      Full
      Equal
      Decorrelated
      Spread(Double)
    } derive(Eq, 
    Debug    )

    How much randomness goes into a delay.

    A retry with no jitter makes every client that failed together try again together, which is how one outage becomes a second one. Full, Equal and Decorrelated are the strategies AWS's "Exponential Backoff and Jitter" measures; Spread is the randomisation factor of Google's client libraries and the one gRPC retries with:

    Rigidthe computed delay, unchanged. Reproducible, and it synchronises clients
    Fulluniform over [0, cap]. The one that spreads clients furthest
    Equalhalf the cap plus uniform over [0, cap/2]. Keeps a floor under the wait
    Decorrelateduniform over [base, previous × 3]. Grows from where it last landed rather than from the attempt number
    Spread(f)uniform over [cap × (1 − f), cap × (1 + f)], f held to [0, 1]. So the wait can pass the cap by f

    Jitter::equal

    source
    fn Jitter::equal(Jitter, Jitter) -> Bool

    Jitter::not_equal

    source
    fn Jitter::not_equal(x : Jitter, y : Jitter) -> Bool

    Jitter::to_repr

    source

    Outcome

    source
    pub(all) enum Outcome {
      Success
      Failure
      Drop
    } derive(Eq, 
    Debug    )

    How a call ended, as a breaker records it.

    A Drop is a call refused before it ran: traffic, but neither a success nor a failure of whatever is downstream.

    Outcome::equal

    source
    fn Outcome::equal(Outcome, Outcome) -> Bool

    Outcome::not_equal

    source
    fn Outcome::not_equal(x : Outcome, y : Outcome) -> Bool

    Outcome::to_repr

    source

    Period

    source
    pub struct Period {
      quota : Int
      period : Int64
      align : Bool
      start : Int64?
      count : Int
    }

    A fixed window counter: at most quota events per period, the count starting again when a period ends.

    Cheaper than [Sliding], which keeps every stamp, and coarser: a client can spend one quota at the end of a period and another at the start of the next. That is the trade go-zero's PeriodLimit makes, and this is it without Redis.

    A period opens on the first event after the last one closed. With align it opens on a multiple of period on the caller's clock instead, so with a Unix clock a daily quota resets at midnight UTC rather than a day after first use.

    Period::left

    source
    fn Period::left(self : Period, now : Int64) -> Int

    How many events remain in the current period.

    Period::new

    source
    fn Period::new(quota : Int, period : 
    Span    , align? : Bool) -> Period

    A counter no event has touched. A period shorter than a nanosecond is raised to one.

    Period::next

    source
    fn Period::next(self : Period, now : Int64) -> 
    Span

    How long until the current period ends. Zero when no period is open, because the next event would open one.

    Period::take

    source
    fn Period::take(self : Period, now : Int64) -> Quota

    Count one event at now and say where the quota stands.

    Promise

    source
    pub struct Promise {
      breaker : Breaker
    }

    A call the breaker let through. Exactly one of pass and fail settles it; until then the window holds no record of how the call went.

    Promise::fail

    source
    fn Promise::fail(self : Promise, now : Int64) -> Unit

    The call failed.

    Promise::pass

    source
    fn Promise::pass(self : Promise, now : Int64) -> Unit

    The call succeeded.

    Quota

    source
    pub(all) enum Quota {
      Under
      Hit
      Over
    } derive(Eq, 
    Debug    )

    Where a [Period] stands after an event: under the quota, exactly on it, or past it.

    The event that hits the quota is still admitted. It is told apart because it is the last one, which is when a caller would warn that the next will be refused. These are go-zero's Allowed, HitQuota and OverQuota.

    Quota::equal

    source
    fn Quota::equal(Quota, Quota) -> Bool

    Quota::not_equal

    source
    fn Quota::not_equal(x : Quota, y : Quota) -> Bool

    Quota::to_repr

    source

    Shed

    source
    pub(all) struct Shed {
      size : Int
      window : 
    Span
      cpu : Int
      latency : 
    Span
      beta : Double
      cool : 
    Span
      floor : Double
    } derive(Eq, 
    Debug    )

    The constants of adaptive load shedding.

    The preset is [shed]. Build another with [Shed::new], or update one in place with { ..shed, cpu: 800 }.

    Shed::equal

    source
    fn Shed::equal(Shed, Shed) -> Bool

    Shed::new

    source
    fn Shed::new(size? : Int, window? : 
    Span    , cpu? : Int, latency? : 
    Span    , beta? : Double, cool? : 
    Span    , floor? : Double) -> Shed

    A policy by name, every knob with the preset's value.

    Shed::not_equal

    source
    fn Shed::not_equal(x : Shed, y : Shed) -> Bool

    Shed::to_repr

    source

    Shedder

    source
    pub struct Shedder {
      shed : Shed
      passes : Window[Total]
      latencies : Window[Total]
      flying : Int64
      average : Double
      hot : Int64?
      dropped : Bool
    }

    Load shedding that refuses work only when the server is both hot and full.

    Hot means the CPU is at or over the threshold, or was within the cooling time and the shedder has refused since. Full means more calls are in flight than the server has shown it can finish: the busiest bucket's passes per millisecond times the fastest bucket's mean response time, scaled down as the CPU nears 100%. Both the moving average and the current count of calls in flight must be over that figure.

    A CPU spike with nothing queued sheds nothing, and neither does a queue on an idle CPU. CPU use is read by the caller, in thousandths, and passed to each [Shedder::allow].

    Shedder::allow

    source
    fn Shedder::allow(self : Shedder, now : Int64, cpu : Int) -> Ticket?

    Ask to start one call at now, with the CPU at cpu thousandths.

    Some means go ahead, and the caller settles the ticket once the call has finished.

    Shedder::flying

    source
    fn Shedder::flying(self : Shedder) -> Int64

    How many calls are in flight.

    Shedder::new

    source
    fn Shedder::new(shed? : Shed, now? : Int64) -> Shedder

    A shedder with empty windows opening at now.

    Sliding

    source
    pub struct Sliding {
      quota : Int
      window : 
    Span
      stamps : Array[Int64]
    }

    A sliding window counter: at most quota events in any window long stretch.

    Where a bucket smooths, a window counts. A quota of "a thousand a day" is a window, not a rate, and answering it with a bucket would let a client spend the whole day's allowance in the first minute.

    The stamps of the events in the window are kept, so the count is exact rather than the approximation a two-bucket scheme gives.

    Sliding::left

    source
    fn Sliding::left(self : Sliding, now : Int64) -> Int

    How many events remain in the current window.

    Sliding::new

    source
    fn Sliding::new(quota : Int, window : 
    Span    ) -> Sliding

    An empty window.

    Sliding::next

    source
    fn Sliding::next(self : Sliding, now : Int64) -> 
    Span

    When the window will next have room, which is when its oldest event falls out.

    Sliding::take

    source
    fn Sliding::take(self : Sliding, now : Int64) -> Bool

    Whether an event is within the quota now, counting it if it is.

    Sre

    source
    pub(all) struct Sre {
      k : Double
      floor : Double
      protection : Double
      probe : 
    Span
      size : Int
      window : 
    Span
    } derive(Eq, 
    Debug    )

    The constants of Google SRE's client-side throttling.

    The preset is [sre]. Build another with [Sre::new], or update one in place with { ..sre, k: 2.0 }.

    Sre::equal

    source
    fn Sre::equal(Sre, Sre) -> Bool

    Sre::new

    source
    fn Sre::new(k? : Double, floor? : Double, protection? : Double, probe? : 
    Span    , size? : Int, window? : 
    Span    ) -> Sre

    A policy by name, every knob with the preset's value.

    Sre::not_equal

    source
    fn Sre::not_equal(x : Sre, y : Sre) -> Bool

    Sre::to_repr

    source

    Tally

    source
    pub struct Tally {
      total : Int64
      success : Int64
      failure : Int64
      drop : Int64
    } derive(Default, Eq, 
    Debug    )

    Outcomes counted in one bucket.

    total counts every call including the dropped ones, so a bucket being throttled still reads as busy rather than idle.

    Tally::add

    source
    fn Tally::add(self : Tally, o : Outcome) -> Unit

    Count one call ending as o.

    Tally::default

    source
    fn Tally::default() -> Tally

    Tally::equal

    source
    fn Tally::equal(Tally, Tally) -> Bool

    Tally::not_equal

    source
    fn Tally::not_equal(x : Tally, y : Tally) -> Bool

    Tally::to_repr

    source

    Ticket

    source
    pub struct Ticket {
      shedder : Shedder
      start : Int64
    }

    A call the shedder let through. Exactly one of pass and fail settles it.

    Ticket::fail

    source
    fn Ticket::fail(self : Ticket) -> Unit

    The call failed. It leaves flight but says nothing about capacity.

    Ticket::pass

    source
    fn Ticket::pass(self : Ticket, now : Int64) -> Unit

    The call succeeded at now. Its response time counts towards what the server has shown it can do.

    Total

    source
    pub struct Total {
      sum : Int64
      count : Int64
    } derive(Default, Eq, 
    Debug    )

    A sum and how many values went into it, which is a mean once count is positive.

    Total::add

    source
    fn Total::add(self : Total, v : Int64) -> Unit

    Add one value.

    Total::default

    source
    fn Total::default() -> Total

    Total::equal

    source
    fn Total::equal(Total, Total) -> Bool

    Total::not_equal

    source
    fn Total::not_equal(x : Total, y : Total) -> Bool

    Total::to_repr

    source

    Window

    source
    pub struct Window[B] {
      width : Int64
      buckets : Array[B]
      head : Int
      start : Int64
    }

    A rolling window: size buckets that together cover the last window of time, oldest dropping off as time moves on.

    The bucket type is the caller's. A breaker keeps a [Tally] of outcomes per bucket, a shedder a [Total] of response times; anything with a Default can be a bucket, and the default is what a bucket is cleared back to.

    Time never moves on its own: every call carries the now it happens at. Writing at a later now clears the buckets the gap swept past; reading at a later now skips them without clearing, so a window nobody writes to still ages out, and two reads at the same now agree.

    Window::each

    source
    fn[B] Window::each(self : Window[B], now : Int64, f : (B) -> Unit, current? : Bool) -> Unit

    Visit the buckets still inside the window at now, oldest first.

    With current false the bucket now falls in is left out: it has seen only part of its time, and a reading that takes the busiest or the fastest bucket should not compare a partial one against whole ones. go-zero's shedder reads its windows this way.

    Window::new

    source
    fn[B : Default] Window::new(size~ : Int, window~ : 
    Span    , now? : Int64) -> Window[B]

    An empty window of size buckets spanning window, its first bucket opening at now.

    A size below one is raised to one, and so is a bucket narrower than a nanosecond.

    Window::size

    source
    fn[B] Window::size(self : Window[B]) -> Int

    How many buckets the window has.

    Window::slot

    source
    fn[B : Default] Window::slot(self : Window[B], now : Int64) -> B

    The bucket now falls in, for the caller to add to.

    The buckets the gap since the last write swept past are cleared first. The head is snapped back to a bucket boundary, so buckets keep their width rather than drifting with the arrival time of each write.

    Window::width

    source
    fn[B] Window::width(self : Window[B]) -> 
    Span

    How much time one bucket covers.

    backoff

    source
    let backoff : Backoff

    The preset: three attempts, a hundred milliseconds doubling to a one-second ceiling, spread by a fifth either way.

    The schedule is the retry policy gRPC's proposal A6 prints — initialBackoff 0.1s, maxBackoff 1s, backoffMultiplier 2 — and the spread is the random(0.8, 1.2) it applies, which grpc-go and grpc-java both draw. Three attempts is the default of the AWS SDKs' standard retry mode.

    shed

    source
    let shed : Shed

    The preset: five seconds in fifty buckets, overloaded at 90% CPU, a one-second assumed response time, an average weighted 0.9, one second to cool and never less than a tenth of capacity.

    These are the constants of go-zero's core/load/adaptiveshedder.go.

    sre

    source
    let sre : Sre

    The preset: k 1.5 decaying to 1.1, five calls of protection, a probe every second, and ten seconds of history in forty buckets.

    These are the constants of go-zero's core/breaker/googlebreaker.go. k and protection are the K and the floor of the SRE book's equation 21-1; the decay, the probe and the recovery scaling are go-zero's additions.

    Source Files

    Powered by MoonBit

    Site sourceReport issuePackagesBuild queueSkillsStatistics

    © 2026 mooncakes.io