🥮 mooncakes.io

    bit_objstore

    Object storage abstraction with compare-and-swap writes (S3/R2/GCS)

    git
    storage
    s3
    object-store
    Download zip
    Author
    Version
    0.47.0
    License
    Apache-2.0
    Repository
    Last updated
    7 hours ago
    Downloads
    69

    Dependencies

    ObjectStore

    source
    pub(open) trait ObjectStore {
      async fn get(Self, String) -> GetOutcome raise 
    GitError
      async fn get_range(Self, String, ByteRange) -> GetOutcome raise 
    GitError
      async fn get_if_none_match(Self, String, String) -> GetOutcome raise 
    GitError
      async fn put(Self, String, Bytes, PutCondition) -> PutOutcome raise 
    GitError
      async fn delete(Self, String) -> Unit raise 
    GitError
      async fn list(Self, String, String) -> ObjListing raise 
    GitError
      fn supports_cas(Self) -> Bool
    }

    The storage contract every backend implements.

    Async-only by design. Every real backend (S3, R2, GCS) is async, and an in-memory or on-disk backend satisfies an async method with a body that never suspends — so one trait keeps callers on a single code path rather than forcing a sync and an async copy of everything above it.

    ByteRange

    source
    pub(all) struct ByteRange {
      start : Int64
      end : Int64
    } derive(Eq, 
    Debug    )

    Inclusive byte range, matching HTTP Range: bytes=<start>-<end>.

    FsStore

    source

    Filesystem backend: one file per key under a root directory.

    Two uses. It lets the WAL be exercised against real IO rather than only a map, and it gives a single-node deployment that needs no bucket at all.

    Single-writer only. Its compare-and-swap is a read-then-write with no lock, so it is safe for one process and not for several. supports_cas answers true because the preconditions really are enforced — against concurrent writers on the same directory, use a real object store.

    FsStore::new

    source

    GetOutcome

    source
    pub(all) enum GetOutcome {
      Found(Bytes, String)
      NotModified
      Missing
    } derive(Eq, 
    Debug    )

    Result of a read.
    impl Show for GetOutcome

    GetOutcome::unwrap

    source
    fn GetOutcome::unwrap(self : GetOutcome, key : String) -> (Bytes, String) raise 
    GitError

    Extract the body of a successful read, raising when the key is absent.

    Convenience for call sites that have already established the key must exist; Missing there means the store is corrupt, not that the caller should branch.

    HttpReply

    source
    pub(all) struct HttpReply {
      status : Int
      headers : Map[String, String]
      body : Bytes
    }

    Response of the injected transport: status, headers, body.

    MemStore

    source
    pub struct MemStore {
      objects : Map[String, Bytes]
      fail_puts_unknown : Int
      silent_puts : Int
      page_size : Int
    }

    In-memory backend.

    Deliberately faithful to S3 rather than convenient: the version tag is derived from content, exactly as an S3 ETag is, so a test that would break against a real bucket breaks here too. It also carries fault injection, because the Unknown write outcome is the one a WAL is most likely to mishandle and the hardest to provoke against a live store.

    MemStore::fail_next_puts

    source
    fn MemStore::fail_next_puts(self : MemStore, n : Int) -> Unit

    Answer the next n writes with Unknown without applying them.

    MemStore::len

    source
    fn MemStore::len(self : MemStore) -> Int

    MemStore::new

    source
    fn MemStore::new() -> MemStore

    MemStore::set_page_size

    source
    fn MemStore::set_page_size(self : MemStore, n : Int) -> Unit

    Force short listing pages so callers must handle continuation.

    MemStore::silence_next_puts

    source
    fn MemStore::silence_next_puts(self : MemStore, n : Int) -> Unit

    Apply the next n writes but answer Unknown, simulating a response lost after the store committed.

    ObjEntry

    source
    pub(all) struct ObjEntry {
      key : String
      size : Int64
      etag : String
    } derive(Eq, 
    Debug    )

    One entry in a listing.

    ObjListing

    source
    pub(all) struct ObjListing {
      entries : Array[ObjEntry]
      next : String?
    } derive(Eq, 
    Debug    )

    A page of a listing. next carries the continuation token when the store truncated the result.

    PutCondition

    source
    pub(all) enum PutCondition {
      Unconditional
      IfNotExists
      IfMatch(String)
    } derive(Eq, 
    Debug    )

    Precondition attached to a write.

    The backing store maps these onto its own primitive: S3 / R2 use If-None-Match: * and If-Match: <etag>; GCS uses ifGenerationMatch=0 and ifGenerationMatch=<generation>.

    PutOutcome

    source
    pub(all) enum PutOutcome {
      Applied(String)
      NotApplied
      Unknown
    } derive(Eq, 
    Debug    )

    Result of a write.

    Unknown is not an error and must not be retried blindly: the write may or may not have landed. The caller re-reads and reconciles. Collapsing it into failure is how a WAL corrupts itself.
    impl Show for PutOutcome

    PutOutcome::is_applied

    source
    fn PutOutcome::is_applied(self : PutOutcome) -> Bool

    Whether a write landed.

    S3Config

    source
    pub(all) struct S3Config {
      endpoint : String
      region : String
      bucket : String
      access_key_id : String
      secret_access_key : String
      session_token : String?
      path_style : Bool
    }

    Connection details for an S3-compatible endpoint.

    S3Store

    source
    pub struct S3Store {
      config : S3Config
      send : async (String, String, Map[String, String], Bytes) -> HttpReply raise 
    GitError
      now : () -> String
      page_size : Int
    }

    S3Store::new

    source
    fn S3Store::new(config : S3Config, send : async (String, String, Map[String, String], Bytes) -> HttpReply raise 
    GitError    , now : () -> String, page_size? : Int) -> S3Store

    SignedRequest

    source
    pub(all) struct SignedRequest {
      verb : String
      url : String
      headers : Map[String, String]
      body : Bytes
    }

    A request ready to hand to an HTTP client.

    civil_from_days

    source
    fn civil_from_days(days : Int64) -> (Int, Int, Int)

    Civil date from days since the Unix epoch (Howard Hinnant's algorithm).

    Valid for the whole proleptic Gregorian range, so a clock that is badly wrong produces a wrong date rather than nonsense.

    endpoint_host

    source
    fn endpoint_host(endpoint : String) -> String raise 
    GitError

    Host (with port, when present) of an endpoint URL.

    format_amz_date

    source
    fn format_amz_date(epoch_seconds : Int64) -> String

    Format Unix epoch seconds as SigV4's YYYYMMDDTHHMMSSZ, in UTC.

    hex_sha256

    source
    fn hex_sha256(data : Bytes) -> String

    Lowercase hex SHA-256 of a payload.

    lex_compare

    source
    fn lex_compare(a : String, b : String) -> Int

    Byte-lexicographic string ordering.

    String::compare orders by length first — it answers that "b" sorts before "aa". That is a fine total order for a local sort, and wrong for any ordering that has to agree with another system's. SigV4 canonical headers, S3 key listings and Git ref advertisements are all specified in byte order, so they need this instead.

    lex_lt

    source
    fn lex_lt(a : String, b : String) -> Bool

    a < b in byte-lexicographic order.

    list_all

    source
    async fn[S : ObjectStore] list_all(store : S, prefix : String) -> Array[ObjEntry] raise 
    GitError

    Read every page of a listing under prefix.

    Callers that need the whole key set (a checkpoint fold, a GC sweep) should use this rather than paging by hand.

    mem_etag

    source
    fn mem_etag(body : Bytes) -> String

    Version tag for a body. Content-derived, like an S3 ETag: writing identical bytes twice yields the same tag, and callers must not assume a tag changes just because a write happened.

    parse_list_objects_v2

    source
    fn parse_list_objects_v2(body : String) -> ObjListing

    Parse a ListObjectsV2 response body.

    Pagination is expressed as start-after rather than a continuation token, so a truncated page reports its last key as the cursor. That keeps the ObjectStore contract to one string and works identically on stores whose continuation tokens are opaque.

    put_immutable

    source
    async fn[S : ObjectStore] put_immutable(store : S, key : String, body : Bytes) -> PutOutcome raise 
    GitError

    Write a content-addressed object, treating "already there" as success.

    Packs and snapshots are named by their own hash, so a create-only PUT that loses the race wrote identical bytes. This is what makes the pre-CAS steps of a push safe to repeat after an Unknown.

    s3_canonical_request

    source
    fn s3_canonical_request(config : S3Config, verb : String, path : String, query : Array[(String, String)], headers : Map[String, String], body : Bytes, amz_date : String) -> String raise 
    GitError

    The canonical request string, exposed for tests and for diagnosing the SignatureDoesNotMatch responses that are otherwise opaque.

    sign_s3_request

    source
    fn sign_s3_request(config : S3Config, verb : String, path : String, query : Array[(String, String)], headers : Map[String, String], body : Bytes, amz_date : String) -> SignedRequest raise 
    GitError

    Sign a request for an S3-compatible endpoint.

    amz_date is YYYYMMDDTHHMMSSZ; the credential scope's datestamp is its first eight characters. Taking it as an argument rather than reading a clock is what makes this function testable.

    strip_etag_quotes

    source
    fn strip_etag_quotes(etag : String) -> String

    S3 returns ETags wrapped in quotes; this API treats version tags as opaque unquoted strings, so the quotes are stripped on the way in and restored on the way out.

    uri_encode

    source
    fn uri_encode(value : String, encode_slash : Bool) -> String

    RFC 3986 percent-encoding, as SigV4 requires it.

    Only A-Z a-z 0-9 - . _ ~ survive unencoded. / survives too when encode_slash is false, which is how the canonical URI keeps its path separators while a query value does not.

    xml_unescape

    source
    fn xml_unescape(value : String) -> String

    Decode the five XML entities S3 uses when escaping keys.

    Runs between entities are copied as substrings rather than code unit by code unit, so a key containing astral characters keeps its surrogate pairs intact.

    Source Files

    Powered by MoonBit

    Site sourceReport issuePackagesBuild queueSkillsStatistics

    © 2026 mooncakes.io