README- UBackoff
- UBucket
- UGate
- UJitter
- USliding
- ::equalLimits::equal
- ::newLimits::new
- ::not_equalLimits::not_equal
- ::to_reprLimits::to_repr
- ::busyPool::busy
- ::closePool::close
- ::dropPool::drop
- ::evictPool::evict
- ::givePool::give
- ::idlePool::idle
- ::livePool::live
- ::madePool::made
- ::newPool::new
- ::refusalPool::refusal
- ::takePool::take
- ::unmadePool::unmade
- TTaken
- ::newWard::new
- Vbackoff
- Vforever
- Vlimits
- Vpick
- Vward
moonpool
Resource pools, backoff retries and flow control for MoonBit.
pool
connection-pool
circuit-breaker
backoff
moonbit
Author
Version
0.3.1
License
Apache-2.0
Repository
Last updated
yesterday
Downloads
1K
moonpool
let pool : @moonpool.Pool[Conn] = @moonpool.Pool::new(
limits=@moonpool.Limits::new(size=20),
)
let now = 0L
match pool.take(now, roll=0.42) {
Ready(conn) => ignore(pool.give(conn, now)) // lend, then take it back
Stale(conn) => ignore(conn) // too old: close it, ask again
Make => pool.made(1, now) // open one; unmade(now) if that fails
Wait => () // all in use: queue the caller
Later(wait) => ignore(wait) // making is held back this long
Shed => () // fail now: the backend or the host said no
}#Six answers
| Means | What the caller does | |
|---|---|---|
| Ready(conn) | An idle resource, the most recently returned one | Use it, then give it back |
| Stale(conn) | An idle resource past its age | Close it, then take again |
| Make | There is room, and it is reserved | Open one, then made — or unmade if that fails |
| Wait | Every live slot is in use | Queue the caller; the pool has no queue of its own |
| Later(span) | Making is held back | Try again after that span |
| Shed | Refuse | Fail fast |
#The four limits
| Preset | What it decides | |
|---|---|---|
| size | 10 | How many may be live at once: idle, lent, and being made |
| idle | 10 | How many may sit idle; give answers false when that many are already there |
| keep | 30 min | How long an idle resource may sit before it is retired |
| life | forever | How long a resource may live at all, however busy it has been |
Zero means no limit for the two ages. Limits::new takes any of them by name, and
{ ..limits, size: 32 } changes one and keeps the rest.
#Nothing is lost
take hands an idle resource past its age back as Stale rather than skipping it, so a
caller closes it whether or not the eviction timer has run. A resource is recognised on its
way back by identity, not by arrival: give does not reset its birth, so life retires a
connection that has been reused all along. A broken one goes to drop, which is what keeps
the live count honest.
#Around the pool
A ward can stand in front of the backend: a breaker that refuses new connections at the rate
a failing one deserves, a bucket that spaces makes out, backoff after a make that failed, and
a shedder that stops lending while the host is hot. Every part is optional, and a pool with
an empty ward behaves exactly as a pool without one. pick chooses between several pools to
one service: two drawn at random, the one that refuses less, then the one with fewer in use.
#Install
moon add moonbitstack/moonpool#See it run
moon run src/examples/simA synthetic client drives a real pool for twenty-four seconds and prints one row a second:
how many lanes were lent, how many sat idle, and how many callers had to wait.
The same engine, compiled to JavaScript, runs the
simulator — the chart, the boards and the
refusal line are the pool's own answers, not a mock.
#Licence
Apache-2.0. See LICENSE.
Backoff
When to try again, and how long to wait first.
The preset is [backoff]. Build another with [Backoff::new], or update one in
place with { ..backoff, attempts: 5 }.
Bucket
A token bucket: rate tokens accrue per second up to burst, and each request
takes one.
It is the rate limiter that allows a burst after a quiet period, which is what
a limit on a human-facing API wants — nobody should be refused for making two
requests in the same second after an hour of making none.
Time is nanoseconds from the caller's clock, as everywhere in this package.
Gate
A count with a ceiling: the concurrency limit a server puts on itself.
Unlike a rate, this does not depend on time at all — it is how many things are
happening at once. A server past its limit answers rather than queues, which is
what keeps a queue from becoming the outage.
Jitter
How much randomness goes into a delay.
A retry with no jitter makes every client that failed together try again
together, which is how one outage becomes a second one. Full, Equal and
Decorrelated are the strategies AWS's "Exponential Backoff and Jitter"
measures; Spread is the randomisation factor of Google's client libraries and
the one gRPC retries with:
| Rigid | the computed delay, unchanged. Reproducible, and it synchronises clients |
| Full | uniform over [0, cap]. The one that spreads clients furthest |
| Equal | half the cap plus uniform over [0, cap/2]. Keeps a floor under the wait |
| Decorrelated | uniform over [base, previous × 3]. Grows from where it last landed rather than from the attempt number |
| Spread(f) | uniform over [cap × (1 − f), cap × (1 + f)], f held to [0, 1]. So the wait can pass the cap by f |
Sliding
A sliding window counter: at most quota events in any window long stretch.
Where a bucket smooths, a window counts. A quota of "a thousand a day" is a
window, not a rate, and answering it with a bucket would let a client spend the
whole day's allowance in the first minute.
The stamps of the events in the window are kept, so the count is exact rather
than the approximation a two-bucket scheme gives.
Limits
The limits a pool holds itself to.
The preset is [limits]. Build another with [Limits::new], or update one in
place with { ..limits, size: 32 }.
Limits::new
Limits by name, every knob with the preset's value.
Pool
type Pool[T]The bookkeeping of holding a limited number of things.
It does not open, close, or wait — a caller with a runtime does all three. What
it knows is how many are live, which are idle, which are lent and since when
each was made, and which have sat or lived long enough to be let go.
Time is passed in as whatever the caller's clock reads, in nanoseconds; the
value's origin does not matter because only differences are used. That is what
makes a test drive a pool through an hour in three lines.
A lent resource is recognised on its way back by identity, which is what every
connection handle has. That is how its birth survives the loan: a pool that
stamped a returned resource as new would never retire one by age, however old.
Idle resources are kept as (item, born, rested), lent ones as (item, born,ticket), and each resource being made as the breaker's promise and the
shedder's ticket it will settle.
Pool::close
Everything the pool holds idle, emptied out for the caller to close — what a
shutdown does. Lent resources are the borrowers' to return; each comes back
through give and can be closed then.
Pool::drop
Say a lent resource is not coming back, because it broke.
This is what keeps the live count honest: without it a failed connection would
hold a slot for ever and the pool would stop making new ones.
Pool::evict
Every idle resource that has sat longer than keep or lived longer than life,
removed from the pool and handed back for the caller to close.
Called on a timer by whoever owns the runtime. take hands a stale one back too,
so a resource is never lost for want of the timer having run.
Pool::give
Hand one back, keeping the birth it was lent with.
false means the pool did not keep it and the caller should close it: there were
already enough idle. A resource that is broken goes to [Pool::drop] instead,
because a pool that kept it would hand it to the next caller.
A resource the pool does not recognise is one made after a Make without
[Pool::made] being called: it fills the room take reserved and is taken as
born now.
Pool::live
How many resources exist right now: idle, lent, and being made.
Pool::made
Register a resource made after take said Make, as lent and born now.
Pool::refusal
The share of new connections the ward's breaker is refusing at now; zero
without a breaker.
Pool::take
Take one, or be told to make one, or to wait — or be handed one that is too old
to use.
The most recently returned resource is handed out first. That is not arbitrary:
a warm connection is likelier to still be good than a cold one, and handing out
the newest lets the oldest age past keep and be evicted, which is how a pool
shrinks after a burst.
Stale is an idle resource past its age. It is already out of the pool and out
of the count; the caller closes it and takes again. Handing it back rather than
dropping it is the point — a pool that discarded it silently would leak whatever
it holds whenever the eviction timer had not run first.
Pool::unmade
Give back a reservation that did not become a resource: the attempt to make
it failed. The breaker counts the failure, and with a Backoff in the ward
no Make is handed out again until its delay has passed; roll draws the
jitter.
Taken
What a pool answers when asked for a resource.
It never blocks and never opens anything: waiting and connecting both need a
runtime, and this package has none. Make is the pool saying there is room for
another, and reserving it; Wait is it saying there is not, and the caller's
queue decides what that means. Stale is an idle resource too old to use, handed
back so the caller can close it before asking again.
The last two come only from a [Ward]: Later is making held back for that
long, by the bucket or by backoff after a failure; Shed is the breaker
refusing new connections to a failing backend, or the shedder refusing to lend
while the process is overloaded.
Ward
backoff
The preset: three attempts, a hundred milliseconds doubling to a one-second
ceiling, spread by a fifth either way.
The schedule is the retry policy gRPC's proposal A6 prints — initialBackoff
0.1s, maxBackoff 1s, backoffMultiplier 2 — and the spread is the
random(0.8, 1.2) it applies, which grpc-go and grpc-java both draw. Three
attempts is the default of the AWS SDKs' standard retry mode.
forever
A span meaning "no limit", which is what zero is taken to mean for the two
ages: a resource with no maximum age is never too old.
limits
The preset: ten live at once, ten of them allowed to sit idle, discarded after
thirty minutes idle and never by age.
Ten live is what most services settle on, and thirty minutes is long enough that
a warm pool survives a quiet spell. Keeping idle equal to size means a pool
that grew under load does not immediately throw away what it just made.
pick
Which of several pools to one service to take from: two drawn by first
and second, each in [0, 1), and of those the one whose breaker refuses
less, then the one with fewer in use.
Two random choices spread load almost as evenly as scanning every pool, and
cost two draws instead of a walk over all of them. -1 when there are no pools.
Source Files
Install
Download zipResource pools, backoff retries and flow control for MoonBit.
poolconnection-poolcircuit-breakerbackoffmoonbit
Apache-2.0
Updated yesterday
1K downloads