Sign in

    sdk-runtime

    API-agnostic client runtime for MoonBit SDKs: error taxonomy, Retry-After and backoff, tri-state JSON fields, open enums. Built on gaato/http; no async runtime dependency.

    sdk
    http
    retry
    rate-limit
    pagination
    json
    Download zip
    Author
    Version
    0.7.0
    License
    Apache-2.0
    Last updated
    3 days ago
    Downloads
    1K

    Dependencies

    #gaato/sdk-runtime

    API-agnostic client runtime for MoonBit SDKs, built on gaato/http: an error taxonomy (SdkError), retry policies over gaato/http/retry rules (RetryPolicy::conservative(), the default, and stainless(), the official OpenAI and Anthropic SDKs' behaviour), a Client that resolves URLs, applies default headers and auth, runs a rate limiter and middleware, classifies responses and retries (never resending a stream body a transport has read), with optional per-attempt and per-call time limits (Timeouts, through an injected @sync.Timer); a window rate limiter, a paginator, tri-state JSON fields (Presence), and open enums (gaato/sdk-runtime/json). The HTTP-generic parts it builds on (retry rules, Retry-After parsing, multipart, randomness) live in gaato/http. No async runtime dependency: time comes through the Clock trait.

    Unofficial and experimental. Source, issues and design notes: https://github.com/gaato/mbt-sdk

    Inject a gate from gaato/http-async to serialize each bucket until its response is reported. An optional global window counts non-exempt admissions across buckets; both timing and synchronization remain injected.

    let limiter = @runtime.WindowLimiter::new(
    @adapter.AsyncClock::new(),
    @runtime.rate_limit_headers(remaining="x-ratelimit-remaining", reset_after_seconds="x-ratelimit-reset-after"),
    gate=() => @adapter.SemaphoreGate::new(),
    global={ limit: 50, window_ms: 1000 },
    )

    Backoff

    using @gaato/http/retry { type Backoff }

    The retry rule types of gaato/http/retry, re-exported so an SDK can write and adjust a policy as @runtime.X without importing that package.

    Decision

    The retry rule types of gaato/http/retry, re-exported so an SDK can write and adjust a policy as @runtime.X without importing that package.

    Jitter

    using @gaato/http/retry { type Jitter }

    The retry rule types of gaato/http/retry, re-exported so an SDK can write and adjust a policy as @runtime.X without importing that package.

    Methods

    using @gaato/http/retry { type Methods }

    The retry rule types of gaato/http/retry, re-exported so an SDK can write and adjust a policy as @runtime.X without importing that package.

    Outcome

    using @gaato/http/retry { type Outcome }

    The retry rule types of gaato/http/retry, re-exported so an SDK can write and adjust a policy as @runtime.X without importing that package.

    RetryAfter

    The retry rule types of gaato/http/retry, re-exported so an SDK can write and adjust a policy as @runtime.X without importing that package.

    RetryAfterCap

    The retry rule types of gaato/http/retry, re-exported so an SDK can write and adjust a policy as @runtime.X without importing that package.

    RetryAfterSource

    The retry rule types of gaato/http/retry, re-exported so an SDK can write and adjust a policy as @runtime.X without importing that package.

    RetryRule

    The retry rule types of gaato/http/retry, re-exported so an SDK can write and adjust a policy as @runtime.X without importing that package.

    StatusMatch

    The retry rule types of gaato/http/retry, re-exported so an SDK can write and adjust a policy as @runtime.X without importing that package.

    StatusRule

    The retry rule types of gaato/http/retry, re-exported so an SDK can write and adjust a policy as @runtime.X without importing that package.

    StopReason

    The retry rule types of gaato/http/retry, re-exported so an SDK can write and adjust a policy as @runtime.X without importing that package.

    RateLimiter

    pub(open) trait RateLimiter {
    async fn acquire(Self, String, global_exempt~ : Bool) -> Unit
    async fn release(Self, String, status~ : Int, headers~ :
    Headers
    ) -> Unit
    }

    A per-bucket admission policy updated from HTTP responses.

    acquire and release come in pairs: the client calls release exactly once for every acquire that returned, with the response's status and headers, or with status 0 and no headers when the attempt produced no response (a transport failure or a cancellation). A limiter that holds per-bucket state across the exchange, such as a gate that serialises one bucket, relies on that pairing; a stateless window limiter ignores the status-0 calls.

    A cancelled task still gets its release, but inside cancellation any further asynchronous work is cut short, so an implementation that talks to another process must recover on its own if that call does not complete (dropping its connection so the peer can clean up, for example).

    RetryDecider

    pub(open) trait RetryDecider {
    fn decide(Self, SdkError, RetryContext) ->
    Decision

    }

    Decides, after a failed attempt, whether the client sends the request again and how long it waits first.

    RetryPolicy is the general-purpose implementation. An API with rules a RetryRule cannot express — a delay read from a response body, a limit recognised by its message — supplies its own implementation to Client::new, and can hand everything else to a RetryPolicy.

    SdkError

    pub(all) suberror SdkError {
    Transport(
    HttpError
    )
    Status(status~ : Int, headers~ :
    Headers
    , body~ : Bytes)
    RateLimited(retry_after_ms~ : Int?, headers~ :
    Headers
    , body~ : Bytes)
    Decode(message~ : String, body~ : Bytes)
    Config(String)
    } derive(
    Debug
    )

    Failures distinguished by transport, HTTP status, decoding, or configuration.

    SdkError::is_retryable

    fn SdkError::is_retryable(self : SdkError) -> Bool

    Reports transient failures, as RetryPolicy::conservative() classifies them; whether the request's method may be retried is the policy's concern. Every transport failure but Protocol is transient, and a status (or a proxy's refusal to open a tunnel) of 408, 409, 429, 500, 502, 503 or 504.

    SdkError::outcome

    The error as a RetryRule sees it: a response for Status and RateLimited (status 429), a failure for Transport, and None for Decode and Config, which no rule retries.

    test {
    let error = @runtime.RateLimited(
    retry_after_ms=None,
    headers=@http.Headers::new(),
    body=b"",
    )
    guard error.outcome() is Some(Response(status=429, ..)) else {
    fail("expected a 429 response")
    }
    assert_true(@runtime.Config("bad").outcome() is None)
    }

    SdkError::status

    fn SdkError::status(self : SdkError) -> Int?

    Returns an HTTP status only for failures with an HTTP status.

    SdkError::to_repr

    Formats the classified failure for debugging.

    Attempt

    pub(all) struct Attempt {
    bucket : String
    request :
    Request

    attempt : Int
    status : Int
    headers :
    Headers

    body : Bytes
    duration_ms : Int64
    error : SdkError?
    }

    One network attempt of a Client call, as reported to the observer.

    status, headers and body are those of the response (the body is empty for a streaming response, whose body belongs to the caller); an attempt that produced no response (a transport failure or a cancellation) reports status 0, empty headers and body, and the failure in error when there is one. attempt counts from 0, so a request that succeeded after two retries reports its last attempt as 2.

    Auth

    pub(all) enum Auth {
    NoAuth
    Bearer(String)
    Header(String, String)
    }

    Request authentication. Debug output always redacts credential values.
    impl Debug for Auth

    Auth::apply

    Adds authentication only when the request has no header with the same name.

    test {
    let request = @runtime.Bearer("secret").apply(@http.Request::get("/"))
    assert_eq(request.headers.get("authorization"), Some("Bearer secret"))
    }

    Auth::to_repr

    Redacted debug representation of authentication settings.

    Client

    pub struct Client {
    // private fields
    }

    A transport-independent SDK client with authentication, retry, and limits.

    Client::new

    fn Client::new(transport : &
    Transport
    , clock : &
    Clock
    , base_url~ : String, default_headers? :
    Headers
    , auth? : Auth, retry? : &RetryDecider, limiter? : &RateLimiter, middleware? : Array[async (
    Request
    , async (
    Request
    ) ->
    Response
    raise
    HttpError
    ) ->
    Response
    raise
    HttpError
    ], observer? : (Attempt) -> Unit, random? : () -> Double, timeouts? : Timeouts) -> Client

    Creates a client from injected transport and clock implementations.

    retry decides after each failed attempt whether to send again; the default is RetryPolicy::default(). observer is called once per attempt, after the limiter's release and before that decision, including for an attempt that fails or is cancelled; it must not raise and should return promptly. timeouts limits each attempt and each call (see Timeouts); without it nothing is limited here, and only the transport's own timeouts apply.

    Client::send

    async fn Client::send(self : Client, request :
    Request
    , bucket? : String, global_exempt? : Bool, body? : () -> Bytes, accept? : (
    Response
    ) -> Unit raise SdkError) ->
    Response
    raise SdkError

    Sends a buffered request, retrying only as directed by the retry decider. A request whose stream body (Request::body_stream) a transport has read is never sent again, whatever the decider says: the failure is raised.

    bucket names the rate-limit bucket the request is admitted through and global_exempt keeps it out of an account-wide limit; both are passed to the limiter as they are.

    body, when given, replaces the request's body: it is called once per attempt, after the limiter admits that attempt, so a request waiting for admission holds no encoded body (a multipart upload can keep only the segments from @multipart.encode_segments and join them here). The retry decider sees the request without it; the observer sees what was sent.

    accept, when given, inspects each successful (2xx) response and may raise to reject it, for an API that reports failures such as rate limits in a successful response. It runs inside the attempt, after the observer has seen the response: what it raises is retried or raised as the retry decider says, exactly like an error status.

    Client::send_json

    async fn Client::send_json(self : Client, request :
    Request
    , bucket? : String, global_exempt? : Bool, body? : () -> Bytes, accept? : (
    Response
    ) -> Unit raise SdkError) -> Json raise SdkError

    Sends and decodes JSON; an empty successful body is JSON null. body and accept are as for send.

    Client::send_stream

    async fn Client::send_stream(self : Client, request :
    Request
    , bucket? : String, global_exempt? : Bool, body? : () -> Bytes) -> (
    ResponseHead
    , &
    BodyStream
    ) raise SdkError

    Sends a streaming request. A successful body is returned to the caller; an unsuccessful body is read up to 1 MiB, closed, classified, and retried. body, and a stream request body, are as for send.

    GlobalWindow

    pub(all) struct GlobalWindow {
    limit : Int
    window_ms : Int
    } derive(Eq,
    Debug
    )

    An account-wide admission window counted across buckets: at most limit admissions in each window_ms. Both must be positive; a window with a non-positive limit or length imposes no limit, since it would otherwise block forever or reset on every admission.

    GlobalWindow::equal

    Equality for global windows.

    GlobalWindow::not_equal

    fn GlobalWindow::not_equal(x : GlobalWindow, y : GlobalWindow) -> Bool

    Equality for global windows.

    GlobalWindow::to_repr

    Debug representation of global windows.

    LimitScope

    pub(all) enum LimitScope {
    Bucket
    Global
    Shared
    Ignore
    } derive(Eq,
    Debug
    )

    What a 429 response does to a WindowLimiter. Each case applies the Retry-After delay (or the configured default) differently.

    LimitScope::equal

    fn LimitScope::equal(LimitScope, LimitScope) -> Bool

    Equality for limit scopes.

    LimitScope::not_equal

    fn LimitScope::not_equal(x : LimitScope, y : LimitScope) -> Bool

    Equality for limit scopes.

    LimitScope::to_repr

    Debug representation of limit scopes.

    NoLimiter

    pub struct NoLimiter {
    }

    A limiter that never delays requests and ignores releases.

    NoLimiter::acquire

    async fn NoLimiter::acquire(_self : NoLimiter, _bucket : String, global_exempt~ : Bool) -> Unit

    Explicit no-op limiter methods.

    NoLimiter::new

    fn NoLimiter::new() -> NoLimiter

    Creates a limiter that admits every request.

    NoLimiter::release

    async fn NoLimiter::release(_self : NoLimiter, _bucket : String, status~ : Int, headers~ :
    Headers
    ) -> Unit

    Explicit no-op limiter methods.

    Pace

    pub(all) struct Pace {
    group : String
    min_interval_ms : Int
    } derive(Eq,
    Debug
    )

    How far apart one group's admissions must be. WindowLimiter::new(pace?) maps each bucket to its group; admissions in a group start at least min_interval_ms apart, whatever the server's window says. A non-positive interval imposes no spacing.

    Pace::equal

    fn Pace::equal(Pace, Pace) -> Bool

    Equality for paces.

    Pace::not_equal

    fn Pace::not_equal(x : Pace, y : Pace) -> Bool

    Equality for paces.

    Pace::to_repr

    Debug representation of paces.

    Page

    pub(all) struct Page[T] {
    items : Array[T]
    next : String?
    }

    One decoded page and its optional cursor for the following page, for Paginator::with_cursor.

    Paginator

    pub struct Paginator[T, E] {
    // private fields
    }

    A stateful async paginator.

    E is the error the fetch raises, so an SDK exposes its own error type through the paginator instead of SdkError.

    Paginator::collect

    async fn[T, E : Error] Paginator::collect(self : Paginator[T, E], max~ : Int) -> Array[T] raise E

    Collects at most max remaining items. The unused rest of the last page is kept for the next call, so nothing fetched is lost. A non-positive maximum performs no requests and returns an empty array.

    Paginator::each

    async fn[T, E : Error] Paginator::each(self : Paginator[T, E], f : async (T) -> Unit raise E) -> Unit raise E

    Visits every remaining item in page order.

    Paginator::new

    fn[T, E : Error] Paginator::new(fetch : async () -> Array[T]? raise E) -> Paginator[T, E]

    Creates a paginator over a fetch that owns its own cursor.

    fetch returns the next page, or None once the sequence has ended; an empty page ends it as well. The fetch keeps whatever cursor the API uses (a snowflake, a timestamp, a URL) and must advance it only after a page has arrived, so that a fetch that raised is asked for the same page again. For an API whose cursor is one string, with_cursor keeps that state here.

    Paginator::next_page

    async fn[T, E : Error] Paginator::next_page(self : Paginator[T, E]) -> Array[T]? raise E

    Fetches the next non-empty page, or None once the sequence has ended. Items that a previous collect did not use come back first.

    Paginator::with_cursor

    fn[T, E : Error] Paginator::with_cursor(fetch : async (String?) -> Page[T] raise E) -> Paginator[T, E]

    Creates a paginator over a string cursor. The first fetch receives None; each page names the cursor of the following one, and a page without a cursor is the last. A fetch that raised leaves the cursor unchanged.

    RateLimitState

    pub(all) struct RateLimitState {
    remaining : Int
    reset_at_unix_ms : Int64
    } derive(Eq,
    Debug
    )

    Remaining requests and the Unix millisecond instant when a window resets.

    RateLimitState::equal

    Equality for rate-limit states.

    RateLimitState::not_equal

    fn RateLimitState::not_equal(x : RateLimitState, y : RateLimitState) -> Bool

    Equality for rate-limit states.

    RateLimitState::to_repr

    Debug representation of rate-limit states.

    RetryContext

    pub struct RetryContext {
    bucket : String
    request :
    Request

    attempt : Int
    random : Double
    now_unix_ms : Int64
    remaining_ms : Int64?
    replayable : Bool
    }

    What a RetryDecider knows about the attempt that failed, besides the error.

    Client asks a decider only when it may send the request again, so the replayable it passes is always true; the field is there for a decider that hands the context on to RetryRule::decide, and for contexts made with RetryContext::new.

    RetryContext::new

    fn RetryContext::new(request :
    Request
    , bucket? : String, attempt? : Int, random? : Double, now_unix_ms? : Int64, remaining_ms? : Int64, replayable? : Bool) -> RetryContext

    A retry context, for tests and for deciders that consult another. By default the bucket is "", the attempt 0, random 0.5 (the middle of any jitter), the time 0, no total timeout, and replayable is request.body.is_replayable().

    test {
    let context = @runtime.RetryContext::new(@http.Request::get("/"))
    assert_eq(context.attempt, 0)
    assert_true(context.replayable)
    }

    RetryPolicy

    pub(all) struct RetryPolicy {
    rule :
    RetryRule

    } derive(Eq,
    Debug
    )

    A retry policy for Client: a RetryRule of gaato/http/retry applied to SdkErrors.

    Start from a preset — conservative() (the default), stainless(cap~), none() — or from any rule with from_rule, and adjust it with with_max_retries, allow_non_idempotent or a struct update of rule.

    test {
    let policy = @runtime.RetryPolicy::conservative().with_max_retries(3)
    assert_eq(policy.rule.max_retries, 3)
    let post = @http.Request::post("/items")
    let busy = @runtime.Status(status=503, headers=@http.Headers::new(), body=b"")
    let context = @runtime.RetryContext::new(post, random=0.5)
    assert_eq(policy.decide(busy, context), Stop(MethodNotAllowed))
    assert_eq(policy.allow_non_idempotent().decide(busy, context), Wait(500))
    }

    RetryPolicy::allow_non_idempotent

    fn RetryPolicy::allow_non_idempotent(self : RetryPolicy) -> RetryPolicy

    This policy with every Idempotent method set widened to Any, in its status rules and for failures after sending: what retry_non_idempotent:true did before 0.7.0. A retried POST may then run twice.

    test {
    let policy = @runtime.RetryPolicy::conservative().allow_non_idempotent()
    assert_eq(policy.rule.sent_failures, Any)
    assert_true(policy.rule.statuses.iter().all(rule => rule.methods == Any))
    }

    RetryPolicy::conservative

    fn RetryPolicy::conservative() -> RetryPolicy

    The policy Client uses unless told otherwise: two retries; 429 for any method; 408, 409, 500, 502, 503 and 504 only for an idempotent request (@http.is_idempotent, which accepts an Idempotency-Key); a failure that proves nothing was sent (HttpError::is_unsent) for any method, and a proxy's refusal only when its status is one of those; a timeout or network failure after sending only for an idempotent request.

    The wait is retry-after-ms, else Retry-After (seconds or an HTTP date), and a server that asks for more than 60 s is not retried; Retry-After: 0 retries at once. Without one, the backoff starts at 500 ms, doubles up to 30 s and is spread by ±20 %.

    test {
    let policy = @runtime.RetryPolicy::conservative()
    assert_eq(policy.rule.max_retries, 2)
    assert_eq(policy.rule.backoff.delay_ms(1, 0.5), 1000)
    assert_eq(@runtime.RetryPolicy::default(), policy)
    }

    RetryPolicy::decide

    The policy's decision as a dot-callable method.

    RetryPolicy::default

    fn RetryPolicy::default() -> RetryPolicy

    conservative().

    RetryPolicy::equal

    fn RetryPolicy::equal(RetryPolicy, RetryPolicy) -> Bool

    Equality for retry policy values.

    RetryPolicy::from_rule

    A policy applying rule as it is.

    RetryPolicy::none

    fn RetryPolicy::none() -> RetryPolicy

    conservative() without retries.

    RetryPolicy::not_equal

    fn RetryPolicy::not_equal(x : RetryPolicy, y : RetryPolicy) -> Bool

    Equality for retry policy values.

    RetryPolicy::stainless

    The retries of the official Stainless-generated SDKs (openai-python and anthropic-sdk-python, _base_client.py): two retries; 408, 409, 429 and every 5xx for any method, as is any transport failure but a protocol error; x-should-retry: true retries any response and false none.

    The wait is retry-after-ms, else Retry-After; cap decides what a long one does (the OpenAI SDK gives up above two minutes, StopAbove(120000); the Anthropic SDK waits whatever it is told, NoCap), and 0 means no delay was given. Without one, the backoff starts at 500 ms, doubles up to 8 s and is shortened by up to 25 %.

    Retrying a POST after a 5xx or a timeout can run it twice; the official SDKs accept that for model APIs.

    test {
    let policy = @runtime.RetryPolicy::stainless(cap=StopAbove(120000))
    let post = @http.Request::post("/v1/responses")
    let error = @runtime.Status(
    status=500,
    headers=@http.Headers::new(),
    body=b"",
    )
    let context = @runtime.RetryContext::new(post, random=0.0)
    assert_eq(policy.decide(error, context), Wait(375))
    }

    RetryPolicy::to_repr

    Debug representation of retry policy values.

    RetryPolicy::with_max_retries

    fn RetryPolicy::with_max_retries(self : RetryPolicy, max_retries : Int) -> RetryPolicy

    This policy with max_retries retries after the first attempt.

    Timeouts

    pub(all) struct Timeouts {
    timer : &
    Timer

    attempt_ms : Int?
    total_ms : Int?
    }

    Time limits for Client calls, enforced with an injected @sync.Timer (such as gaato/http-async's ClockTimer), since this module has no async runtime of its own.

    attempt_ms bounds each attempt's exchange with the transport: for send and send_json the whole response, for send_stream the response head only (the body belongs to the caller). An exchange that runs out fails with Transport(Timeout(phase=Unknown, ms=<its limit>)), which is released, observed and given to the retry decider like any transport failure; under the presets a timeout is a failure after sending, so conservative() retries it only for an idempotent request.

    total_ms bounds a whole call from when it starts, retries and their waits included. Each exchange is also limited to the time left, the decider is told the time left (RetryContext::remaining_ms) so a RetryPolicy stops instead of waiting past the end (Budget), and an attempt that would start with no time left fails with Transport(Timeout(phase=Unknown, ms=total_ms)) without asking the decider. Waiting for the rate limiter counts against the total but is not interrupted, so that every admission is paired with its release; a limiter wait can therefore overrun the total by up to its own length.

    A limit that is not positive fails at once: attempt_ms every attempt, total_ms the call before anything is sent.

    Timeouts::new

    fn Timeouts::new(timer : &
    Timer
    , attempt_ms? : Int, total_ms? : Int) -> Timeouts

    Time limits on timer; an omitted limit does not apply.

    WindowLimiter

    pub struct WindowLimiter {
    // private fields
    }

    A mutable per-bucket fixed-window limiter driven by an injected clock.

    WindowLimiter::acquire

    async fn WindowLimiter::acquire(self : WindowLimiter, bucket : String, global_exempt~ : Bool) -> Unit

    Explicit fixed-window limiter methods.

    WindowLimiter::new

    fn WindowLimiter::new(clock : &
    Clock
    , parse : (
    Headers
    , Int64) -> RateLimitState?, gate? : () -> &
    Gate
    , pace? : (String) -> Pace, server_key? : (String,
    Headers
    ) -> String?, scope? : (Int,
    Headers
    ) -> LimitScope, global? : GlobalWindow, global_block? : (
    Headers
    ) -> Bool, default_retry_after_ms? : Int) -> WindowLimiter

    Creates a limiter using an API-specific response-header parser.

    pace maps a bucket to its pace group, whose admissions start at least min_interval_ms apart; by default each bucket is its own group with no spacing. It must be pure: acquire and release each call it and must get the same group for the same bucket. gate creates a pace group's gate, held from acquire until the paired release, so the buckets of one group are serialised together; a group nobody holds or waits on forgets its gate and gets a new one on its next acquire. server_key reads, from a response's headers, the key under which the server counts the bucket (several buckets may share one server window). Once it names a key, the bucket's state is recorded and looked up under that key until a later response names another one or the key's state expires; the gate and the pace group stay the bucket's own. scope says what a 429 does (LimitScope); it is asked only about 429 responses, with their status and headers. By default a 429 empties its bucket for its Retry-After, or blocks every bucket when global_block says the 429 is account-wide. global_block is kept for existing callers and is ignored when scope is given. global limits non-exempt admissions across buckets; one with a non-positive limit or length is ignored. default_retry_after_ms supplies a delay when a 429 has no Retry-After. Negative fallback delays and pace intervals are treated as zero. A response's parsed reset and Retry-After count only up to one hour ahead, so no server can hold a bucket, or every request, longer than that.

    WindowLimiter::release

    async fn WindowLimiter::release(self : WindowLimiter, bucket : String, status~ : Int, headers~ :
    Headers
    ) -> Unit

    Explicit fixed-window limiter methods.

    WindowLimiter::state

    fn WindowLimiter::state(self : WindowLimiter, bucket : String) -> RateLimitState?

    Returns the currently recorded state for a bucket, looked up under the key its last response named through server_key, if any.

    classify

    fn classify(response :
    Response
    , now_unix_ms? : Int64) ->
    Response
    raise SdkError

    Returns successful responses unchanged and classifies all other statuses. The error keeps at most the first 1 MiB of the body, as a streaming error body is read; the status and headers are kept whole.

    test {
    let response : @http.Response = {
    status: 204,
    headers: @http.Headers::new(),
    body: b"",
    }
    assert_eq(@runtime.classify(response), response)
    let error = @runtime.Transport(@http.Timeout(phase=Head, ms=500))
    assert_eq(error.status(), None)
    assert_true(error.is_retryable())
    }

    rate_limit_headers

    fn rate_limit_headers(remaining~ : String, reset_after_seconds? : String, reset_unix_seconds? : String) -> ((
    Headers
    , Int64) -> RateLimitState?)

    Builds a parser for conventional remaining/reset response headers.