moonpool

    Resource pools, backoff retries and flow control for MoonBit.

    pool
    resource-pool
    connection-pool
    object-pool
    backoff
    retry
    exponential-backoff
    flow-control
    circuit-breaker
    limits
    load-balancing
    round-robin
    power-of-two-choices
    weighted
    metrics
    observability
    leak-detection
    adaptive-concurrency
    concurrency-limit
    reproducible
    moonbit
    Download zip
    Version
    0.4.0
    License
    Apache-2.0
    Last updated
    11 hours ago
    Downloads
    3K
    moonpool


    // Here a connection is just its number; any type works.
    let pool : @moonpool.Pool[Int] = @moonpool.Pool::new(
    limits=@moonpool.Limits::new(size=20),
    )
    let now = 0L
    match pool.take(now, roll=0.42) {
    Ready(conn) => ignore(pool.give(conn, now)) // lend, then take it back
    Stale(conn) => ignore(conn) // too old: close it, ask again
    Make => pool.made(1, now) // open one; unmade(now) if that fails
    Wait => () // all in use: queue the caller
    Later(wait) => ignore(wait) // making is held back this long
    Shed => () // fail now: the backend or the host said no
    }

    #Six answers

    MeansWhat the caller does
    Ready(conn)An idle resource, the most recently returned oneUse it, then give it back
    Stale(conn)An idle resource past its ageClose it, then take again
    MakeThere is room, and it is reservedOpen one, then made — or unmade if that fails
    WaitEvery live slot is in useQueue the caller; the pool has no queue of its own
    Later(span)Making is held backTry again after that span
    ShedRefuseFail fast

    #The four limits

    PresetWhat it decides
    size10How many may be live at once: idle, lent, and being made
    idle10How many may sit idle; give answers false when that many are already there
    keep30 minHow long an idle resource may sit before it is retired
    lifeforeverHow long a resource may live at all, however busy it has been

    Zero means no limit for the two ages. Limits::new takes any of them by name, and { ..limits, size: 32 } changes one and keeps the rest.

    #Nothing is lost

    take hands an idle resource past its age back as Stale rather than skipping it, so a caller closes it whether or not the eviction timer has run. A resource is recognised on its way back by identity, not by arrival: give does not reset its birth, so life retires a connection that has been reused all along. A broken one goes to drop, which is what keeps the live count honest.

    #The books

    live, idle and busy say what the pool holds now. books says that and what it has done, as counters that only rise — so two readings say what happened between them.

    let counted : @moonpool.Pool[Int] = @moonpool.Pool::new(
    limits=@moonpool.Limits::new(size=8),
    )
    ignore(counted.take(0L))
    counted.made(1, 0L)
    ignore(counted.give(1, 100_000_000L))
    ignore(counted.take(200_000_000L))
    let books = counted.books()
    assert_eq(books.live, 1)
    assert_eq(books.loans, 2L)
    assert_eq(books.reused, 1L)
    // One was made for the asker; the other came out of idle.
    assert_eq(books.made, 1L)

    loans answers the other half: not how many are out, but which, and for how long. How long a caller's work may take is the caller's to say, so the pool only reports.

    for loan in counted.loans(500_000_000L) {
    assert_eq(loan.item, 1)
    assert_eq(loan.held, @moondate.Span::new(millis=300L))
    }

    #Around the pool

    A ward can stand in front of the backend: a breaker that refuses new connections at the rate a failing one deserves, a bucket that spaces makes out, backoff after a make that failed, a shedder that stops lending while the host is hot, and a limiter that holds a lease for every loan and moves its ceiling with how long they take. Every part is optional, and a pool with an empty ward behaves exactly as a pool without one. pick chooses between several pools to one service: two drawn at random, the one that refuses less, then the one with fewer in use. RoundRobin and Weighted choose between targets with no pools behind them, by turns and by share.

    #Conformance

    A pool has no standard to conform to, so its decisions are compared with the answers of two implementations written by other people, in other languages. The same steps — take at a given moment, hand back, give up on a broken one — are run against each of them and against this package, and every hand-out has to be the same resource, with the same number idle and live after it.

    ReferenceWhat it ranHand-outs comparedResult
    SQLAlchemy 2.0.44 QueuePoolfour plans, thirty traces233all: same resource, same counts
    node-generic-pool 3.9.0four plans, thirty traces, ten traces with a dropped resource301all: same resource, same counts

    The four plans pin one decision each: reuse order, the ceiling, the two ages, a broken resource. The traces go after the combinations no plan thinks of — every legal ordering of take and give over one to four seats, drawn from a fixed seed, twelve moves each, with one ask past the ceiling so that a refusal is part of the recording. They carry no clock, so they are the same on every run and on every machine.

    Both references are configured to the policy this package holds by default: a hard ceiling that blocks rather than overflows, last-in-first-out reuse, and one age limit each — the first retires by lifetime, the second by idle time. Three differences are deliberate:

    • A broken resource. SQLAlchemy marks a discarded connection for replacement at the next borrow instead of taking it off its books, so what it hands out next, and how many it says are idle at that instant, are its own bookkeeping rather than the decision. The traces that drop one are therefore compared against node-generic-pool, which discards at once and agrees with this package; SQLAlchemy still runs the four plans, the broken-resource one among them, and agrees on those.
    • A ceiling on idle resources. Neither reference has one, so it is covered by unit tests.

    The steps and the recordings are in scripts/reference/ and src/reference_vectors_wbtest.mbt, and CI runs the references again and fails when the recording no longer matches what they answer.

    #One owner

    A Pool holds mutable state and no lock: it belongs to one task. Several tasks sharing one pool take it under a mutex — @async.Mutex — and everything a task does with an answer (made, give, drop) happens inside that same lock. Nothing here assumes another thread, so nothing has to be made safe twice: the lock above is the only one.

    #Install

    moon add moonbitstack/moonpool

    #See it run

    moon run src/examples/sim

    A synthetic client drives a real pool for twenty-four seconds and prints one row a second: how many lanes were lent, how many sat idle, and how many callers had to wait.

    The same engine, compiled to JavaScript, runs the simulator — the chart, the boards and the refusal line are the pool's own answers, not a mock.

    #Licence

    Apache-2.0. See LICENSE.

    Backoff

    When to try again, and how long to wait first.

    The preset is [backoff]. Build another with [Backoff::new], or update one in place with { ..backoff, attempts: 5 }.

    Bucket

    A token bucket: rate tokens accrue per second up to burst, and each request takes one.

    It is the rate limiter that allows a burst after a quiet period, which is what a limit on a human-facing API wants — nobody should be refused for making two requests in the same second after an hour of making none.

    Time is nanoseconds from the caller's clock, as everywhere in this package.

    Gate

    A count with a ceiling: the concurrency limit a server puts on itself.

    Unlike a rate, this does not depend on time at all — it is how many things are happening at once. A server past its limit answers rather than queues, which is what keeps a queue from becoming the outage.

    Jitter

    How much randomness goes into a delay.

    A retry with no jitter makes every client that failed together try again together, which is how one outage becomes a second one. Full, Equal and Decorrelated are the three shapes the literature names and measures; Spread keeps the wait within a fraction of the ceiling instead, which is the easier one to reason about when the schedule is fixed by something else:

    Rigidthe computed delay, unchanged. Reproducible, and it synchronises clients
    Fulluniform over [0, cap]. The one that spreads clients furthest
    Equalhalf the cap plus uniform over [0, cap/2]. Keeps a floor under the wait
    Decorrelateduniform over [base, previous × 3]. Grows from where it last landed rather than from the attempt number
    Spread(f)uniform over [cap × (1 − f), cap × (1 + f)], f held to [0, 1]. So the wait can pass the cap by f

    Sliding

    A sliding window counter: at most quota events in any window long stretch.

    Where a bucket smooths, a window counts. A quota of "a thousand a day" is a window, not a rate, and answering it with a bucket would let a client spend the whole day's allowance in the first minute.

    The stamps of the events in the window are kept, so the count is exact rather than the approximation a two-bucket scheme gives.

    Books

    pub(all) struct Books {
    live : Int
    idle : Int
    busy : Int
    made : Int64
    failed : Int64
    loans : Int64
    reused : Int64
    retired : Int64
    waited : Int64
    deferred : Int64
    shed : Int64
    } derive(Eq,
    Debug
    )

    What a pool has done, as one reading: how much it holds now, and how much has passed through it.

    The three gauges are the pool at this instant. The counters only ever rise, so the difference between two readings is what happened in between; a service that exports them to its metrics endpoint gets the same numbers a connection pool's management bean would offer.

    loans is the total of resources handed to callers, and reused the part of it taken from idle — so loans - reused is how many were made for the asker, which is the pool's cold-start cost.

    Books::equal

    fn Books::equal(Books, Books) -> Bool

    Books::not_equal

    fn Books::not_equal(x : Books, y : Books) -> Bool

    Books::to_repr

    Lent

    pub(all) struct Lent[T] {
    item : T
    held :
    Span

    } derive(
    Debug
    )

    A resource that is out on loan: what it is, and how long it has been gone.

    A caller compares held with its own threshold to find a loan that was never given back — what a leak detector logs, with the item so the borrower can be traced.

    Limits

    pub(all) struct Limits {
    size : Int
    idle : Int
    keep :
    Span

    life :
    Span

    } derive(Eq,
    Debug
    )

    The limits a pool holds itself to.

    The preset is [limits]. Build another with [Limits::new], or update one in place with { ..limits, size: 32 }.

    Zero means no ceiling, for size, for idle and for both ages. A caller that wants nothing kept idle closes what it borrowed instead of giving it back.

    Limits::equal

    fn Limits::equal(Limits, Limits) -> Bool

    Limits::new

    fn Limits::new(size? : Int, idle? : Int, keep? :
    Span
    , life? :
    Span
    ) -> Limits

    Limits by name, every knob with the preset's value.

    Limits::not_equal

    fn Limits::not_equal(x : Limits, y : Limits) -> Bool

    Limits::to_repr

    Pool

    type Pool[T]

    The bookkeeping of holding a limited number of things.

    It does not open, close, or wait — a caller with a runtime does all three. What it knows is how many are live, which are idle, which are lent and since when each was made, and which have sat or lived long enough to be let go.

    Time is passed in as whatever the caller's clock reads, in nanoseconds; the value's origin does not matter because only differences are used. That is what makes a test drive a pool through an hour in three lines.

    A lent resource is recognised on its way back by identity, which is what every connection handle has. That is how its birth survives the loan: a pool that stamped a returned resource as new would never retire one by age, however old.

    Pool::books

    fn[T] Pool::books(self : Pool[T]) -> Books

    The gauges and counters together, as one reading.

    Pool::busy

    fn[T] Pool::busy(self : Pool[T]) -> Int

    How many are lent out or being made.

    Pool::close

    fn[T] Pool::close(self : Pool[T]) -> Array[T]

    Everything the pool holds idle, emptied out for the caller to close — what a shutdown does. Lent resources are the borrowers' to return; each comes back through give and can be closed then.

    Pool::drop

    fn[T] Pool::drop(self : Pool[T], item : T) -> Unit

    Say a lent resource is not coming back, because it broke.

    This is what keeps the live count honest: without it a failed connection would hold a slot for ever and the pool would stop making new ones.

    Pool::evict

    fn[T] Pool::evict(self : Pool[T], now : Int64) -> Array[T]

    Every idle resource that has sat longer than keep or lived longer than life, removed from the pool and handed back for the caller to close.

    Called on a timer by whoever owns the runtime. take hands a stale one back too, so a resource is never lost for want of the timer having run.

    Pool::give

    fn[T] Pool::give(self : Pool[T], item : T, now : Int64) -> Bool

    Hand one back, keeping the birth it was lent with.

    false means the pool did not keep it and the caller should close it: there were already enough idle. A resource that is broken goes to [Pool::drop] instead, because a pool that kept it would hand it to the next caller.

    A resource the pool does not recognise is one made after a Make without [Pool::made] being called: it fills the room take reserved and is taken as born now.

    One the pool is already holding idle answers false as well: a resource handed back twice would otherwise sit in two seats, and a later take would lend the same handle to two callers.

    Pool::idle

    fn[T] Pool::idle(self : Pool[T]) -> Int

    How many are sitting idle.

    Pool::live

    fn[T] Pool::live(self : Pool[T]) -> Int

    How many resources exist right now: idle, lent, and being made.

    Pool::loans

    fn[T] Pool::loans(self : Pool[T], now : Int64) -> Array[Lent[T]]

    Every resource that is out on loan, with how long it has been out.

    A pool cannot know which loan was forgotten: only the caller knows how long its own work may take. So the pool tells the truth and the caller sets the threshold — anything held longer than that is a leak to log against the item.

    Pool::made

    fn[T] Pool::made(self : Pool[T], item : T, now : Int64) -> Unit

    Register a resource made after take said Make, as lent and born now.

    A handle the pool already holds is not loaned twice: the registration counts, the reservation it came with is settled, and the loan the pool already had stands. One handle lent twice would be two loans to close with one give, which is a slot lost for good.

    Pool::new

    fn[T] Pool::new(limits? : Limits, ward? : Ward) -> Pool[T]

    An empty pool.

    Pool::refusal

    fn[T] Pool::refusal(self : Pool[T], now : Int64) -> Double

    The share of new connections the ward's breaker is refusing at now; zero without a breaker.

    Pool::take

    fn[T] Pool::take(self : Pool[T], now : Int64, roll? : Double, cpu? : Int) -> Taken[T]

    Take one, or be told to make one, or to wait — or be handed one that is too old to use.

    The most recently returned resource is handed out first. That is not arbitrary: a warm connection is likelier to still be good than a cold one, and handing out the newest lets the oldest age past keep and be evicted, which is how a pool shrinks after a burst.

    Stale is an idle resource past its age. It is already out of the pool and out of the count; the caller closes it and takes again. Handing it back rather than dropping it is the point — a pool that discarded it silently would leak whatever it holds whenever the eviction timer had not run first.

    Pool::unmade

    fn[T] Pool::unmade(self : Pool[T], now : Int64, roll? : Double) -> Unit

    Give back a reservation that did not become a resource: the attempt to make it failed. The breaker counts the failure, and with a Backoff in the ward no Make is handed out again until its delay has passed; roll draws the jitter.

    The limiter that let the attempt through is released without a sample: the breaker is what owns a backend that will not connect, and counting the same failure in two places would shrink the pool twice for one cause.

    RoundRobin

    pub struct RoundRobin {
    // private fields
    }

    Takes turns. It holds only a cursor, so it costs nothing to keep one per client.

    RoundRobin::new

    fn RoundRobin::new() -> RoundRobin

    A rotation starting at the first target.

    RoundRobin::pick

    fn RoundRobin::pick(self : RoundRobin, count : Int) -> Int

    The next of count targets, or -1 when there are none. The cursor wraps at the current count, so it stays valid as targets come and go.

    Taken

    pub(all) enum Taken[T] {
    Ready(T)
    Stale(T)
    Make
    Wait
    Later(
    Span
    )
    Shed
    }

    What a pool answers when asked for a resource.

    It never blocks and never opens anything: waiting and connecting both need a runtime, and this package has none. Make is the pool saying there is room for another, and reserving it; Wait is it saying there is not, and the caller's queue decides what that means. Stale is an idle resource too old to use, handed back so the caller can close it before asking again.

    The last two come only from a [Ward]: Later is making held back for that long, by the bucket or by backoff after a failure; Shed is the breaker refusing new connections to a failing backend, or the shedder refusing to lend while the process is overloaded. A ward's limiter can answer Wait as well, when the concurrency it allows is already out on loan.

    Ward

    What stands in front of a pool's backend. Every part is optional and the preset has none, which leaves the pool exactly as it is without a ward.

    The four guards are about a backend that misbehaves. The limiter is about the pool's own concurrency: it holds a lease for every loan, and the limit it keeps moves with how long those loans take, so a quiet backend is allowed more at once and a slow one fewer, under the same size.

    Weighted

    pub struct Weighted {
    // private fields
    }

    Smooth weighted round-robin, nginx's: every pick credits each target with its weight, serves the one with the most credit, and charges it the total. Over a cycle each target gets its share, and the turns interleave: a weight of 5 is not five requests back to back.

    Credit is kept by name, so the list can change order between picks. A target that leaves loses its credit, which was earned under weights that may no longer hold.

    Weighted::new

    fn Weighted::new() -> Weighted

    A balancer with no credit yet.

    Weighted::pick

    fn Weighted::pick(self : Weighted, targets : ArrayView[(String, Int)]) -> Int

    The index of the next of targets, each a name and a weight. A weight of zero or less is never picked, and -1 means nothing can be.

    backoff

    The preset: three attempts, a hundred milliseconds doubling to a one-second ceiling, spread by a fifth either way.

    Each wait is the computed ceiling spread by a fifth below or above it, so a plan of 0.8 to 1.2 of the schedule is easy to explain to whoever asks why the second retry waited 168 ms and not 200.

    forever

    A span meaning "no limit", which is what zero is taken to mean for the two ages: a resource with no maximum age is never too old.

    limits

    let limits : Limits

    The preset: ten live at once, ten of them allowed to sit idle, discarded after thirty minutes idle and never by age.

    Ten live is what most services settle on, and thirty minutes is long enough that a warm pool survives a quiet spell. Keeping idle equal to size means a pool that grew under load does not immediately throw away what it just made.

    pick

    fn[T] pick(pools : ArrayView[Pool[T]], now : Int64, first~ : Double, second~ : Double) -> Int

    Which of several pools to one service to take from: two drawn by first and second, each in [0, 1), and of those the one whose breaker refuses less, then the one with fewer in use.

    Two random choices spread load almost as evenly as scanning every pool, and cost two draws instead of a walk over all of them. -1 when there are no pools.

    ward

    let ward : Ward