🥮 mooncakes.io

    bitx_wal

    Write-ahead log over object storage, coordinated by compare-and-swap

    wal
    storage
    cas
    log
    Download zip
    Author
    Version
    0.47.0
    License
    Apache-2.0
    Repository
    Last updated
    7 hours ago
    Downloads
    1

    Dependencies

    CommitOutcome

    source
    pub(all) enum CommitOutcome {
      Committed(Int64)
      Contended
    } derive(Eq)

    Outcome of a commit.

    Fold

    source
    pub(all) struct Fold {
      body : Bytes
      retained_payloads : Array[String]
    }

    What a fold produced: the snapshot bytes, and the payload keys the snapshot still needs.

    Reporting the retained payloads is the caller's job because only the caller knows which of the folded entries' payloads survived the fold — a Git repack, for instance, replaces several packs with one.

    Wal

    source
    pub struct Wal[S] {
      store : S
      prefix : String
      image : WalImage
      etag : String
      loaded : Bool
      max_attempts : Int
      max_image_bytes : Int
    }

    Wal::append

    source

    Append entries as one atomic commit.

    Several entries in one call are a group commit: they cost a single compare-and-swap between them, which is what keeps concurrent pushes from each paying their own round trip.

    Entries carry payload keys but not payload bytes — write those with put_payload first. That ordering is deliberate: a payload is content addressed and immutable, so writing one that is never referenced leaves garbage, while committing an entry that references a payload not yet written would leave the log describing something that does not exist.

    Wal::compact

    source
    async fn[S : 
    ObjectStore    ] Wal::compact(self : Wal[S], through_seq : Int64, fold : async (Bytes, Array[WalEntry]) -> Fold raise 
    GitError    ) -> CommitOutcome raise 
    GitError

    Fold entries up to and including through_seq into a new snapshot.

    fold receives the previous snapshot (empty on the first compaction) and the entries being folded, and returns the new snapshot. It must be a pure function of those inputs: it may run again after a lost race.

    Returns the sequence the log is now folded through.

    Wal::entries

    source
    fn[S] Wal::entries(self : Wal[S]) -> Array[WalEntry]

    The live entries as of the last sync.

    Wal::get_payload

    source
    async fn[S : 
    ObjectStore    ] Wal::get_payload(self : Wal[S], content_id : String) -> Bytes raise 
    GitError

    Wal::head_seq

    source
    fn[S] Wal::head_seq(self : Wal[S]) -> Int64

    Sequence of the newest entry as of the last sync.

    Wal::image_key

    source
    fn[S] Wal::image_key(self : Wal[S]) -> String

    Wal::new

    source
    fn[S] Wal::new(store : S, prefix : String, max_attempts? : Int, max_image_bytes? : Int) -> Wal[S]

    Wal::payload_key

    source
    fn[S] Wal::payload_key(self : Wal[S], content_id : String) -> String

    Wal::put_payload

    source
    async fn[S : 
    ObjectStore    ] Wal::put_payload(self : Wal[S], body : Bytes) -> String raise 
    GitError

    Store a payload under its content address.

    Safe to repeat: the key is derived from the bytes, so a repeat writes what is already there. Returns the content id to reference from an entry.

    Wal::read_snapshot

    source
    async fn[S : 
    ObjectStore    ] Wal::read_snapshot(self : Wal[S]) -> Bytes raise 
    GitError

    Read the current snapshot, or empty bytes when nothing has been folded.

    Wal::set_settings

    source
    async fn[S : 
    ObjectStore    ] Wal::set_settings(self : Wal[S], settings : String) -> CommitOutcome raise 
    GitError

    Replace the settings blob, under the same compare-and-swap discipline as an append.

    Wal::settings

    source
    fn[S] Wal::settings(self : Wal[S]) -> String

    Wal::snapshot

    source
    fn[S] Wal::snapshot(self : Wal[S]) -> String

    Snapshot key as of the last sync, empty when nothing has been folded.

    Wal::snapshot_key

    source
    fn[S] Wal::snapshot_key(self : Wal[S], content_id : String) -> String

    Wal::sweep_offline

    source
    async fn[S : 
    ObjectStore    ] Wal::sweep_offline(self : Wal[S]) -> Array[String] raise 
    GitError

    Delete unreferenced payloads and snapshots.

    Not safe to run against a log with writers in flight. A payload is written before the entry that references it lands, so between those two moments it looks exactly like garbage. waltier makes the same restriction for the same reason: collection is an offline operation.

    Returns the keys removed.

    Wal::sync

    source

    Refresh the cached image.

    After the first read this is a conditional GET: an unchanged log costs one request that transfers nothing, which is what makes a replica cheap to keep fresh.

    Wal::unreferenced_keys

    source
    async fn[S : 
    ObjectStore    ] Wal::unreferenced_keys(self : Wal[S]) -> Array[String] raise 
    GitError

    Keys that a sweep would delete: payloads and snapshots no longer referenced by the image.

    Separated from the deletion itself so a maintainer can report a sweep before performing one, and so the decision is testable without destroying anything.

    WalEntry

    source
    pub(all) struct WalEntry {
      seq : Int64
      kind : String
      payload_keys : Array[String]
      data : String
      idempotency_key : String
    } derive(Eq)

    What an entry records. The engine does not interpret payload_keys or data; it guarantees only their ordering and durability.

    WalEntry::new

    source
    fn WalEntry::new(kind : String, data : String, payload_keys? : Array[String], idempotency_key? : String) -> WalEntry

    WalImage

    source
    pub(all) struct WalImage {
      version : Int
      snapshot : String
      snapshot_seq : Int64
      snapshot_payloads : Array[String]
      entries : Array[WalEntry]
      settings : String
    } derive(Eq)

    The WAL image: the only object whose rewrite makes anything visible.

    WalImage::empty

    source
    fn WalImage::empty() -> WalImage

    WalImage::head_seq

    source
    fn WalImage::head_seq(self : WalImage) -> Int64

    Sequence of the last entry, folded or live.

    WalImage::live_payloads

    source
    fn WalImage::live_payloads(self : WalImage) -> Array[String]

    Every payload key the image still depends on. Anything else under the payload prefix is garbage.

    DEFAULT_MAX_ATTEMPTS

    source
    let DEFAULT_MAX_ATTEMPTS : Int

    How hard append tries before reporting contention. Each attempt costs a read and a conditional write, so this is a cost ceiling as much as a correctness one.

    DEFAULT_MAX_IMAGE_BYTES

    source
    let DEFAULT_MAX_IMAGE_BYTES : Int

    Refuse to write an image larger than this.

    waltier's equivalent default is 64 MiB. An image is rewritten in full on every commit, so its size is the per-commit cost: letting it grow without bound turns every push into a multi-megabyte round trip. Exceeding it means compaction is overdue, which is a caller error rather than a storage failure, so it is reported as one.

    WAL_FORMAT_VERSION

    source
    let WAL_FORMAT_VERSION : Int

    Bumped only for a change an older reader could not safely ignore.

    content_id

    source
    fn content_id(body : Bytes) -> String

    Content address for a blob: the same bytes always get the same key, which is what makes writing one idempotent.

    decode_image

    source
    fn decode_image(body : Bytes) -> WalImage raise 
    GitError

    encode_image

    source
    fn encode_image(image : WalImage) -> Bytes

    Source Files

    Powered by MoonBit

    Site sourceReport issuePackagesBuild queueSkillsStatistics

    © 2026 mooncakes.io