🥮 mooncakes.io

    moonerasure

    Reed-Solomon coding, byte-range parity updates and placement-aware repair planning

    erasure
    reed-solomon
    storage
    recovery
    gf256
    Download zip
    Author
    Version
    0.3.0
    License
    Apache-2.0
    Repository
    Last updated
    7 days ago
    Downloads
    45

    No dependencies

    #MoonErasure

    MoonBit 原生 Reed–Solomon 编码、选择性重建与增量更新库。

    本库面向备份工具、对象存储适配器、块数据更新与分片传输服务，输入是等长数据片、已知丢失位置和更新前后字节，输出是校验片、指定重建片或更新后的校验片。核心计算不依赖 Go/JavaScript 运行时，支持 Native、JS、Wasm 和 Wasm-GC。

    功能范围以 klauspost/reedsolomon v1.14.2 对照表说明；三个可执行验证场景给出输入、断言及成本模型。参考库提供成熟算法功能基准，本库的实现、帧和对象格式独立编写。

    MoonErasure is a MoonBit library for systematic Reed–Solomon erasure coding over GF(256). It turns k data shards into k + m shards and reconstructs a stripe from any k intact shards. The first k shards retain the original padded data. Applications decide where to store, send, authenticate and persist the shards.

    The library supports complete objects, bounded stripes, independent checksum-protected frames, repair plans, range reads, incremental encoding and decoding, a portable single-file bundle format, and health inspection. The code is original MoonBit and runs on wasm, wasm-gc, js and native targets.

    #0.3.0：区间更新与放置感知修复

    • Codec::update_parity_ranges 仅提供变化区间的旧/新字节即可更新校验片，避免读取未修改的数据区域。
    • Codec::plan_costed 从已知完好的分片中选择最低非负成本的恢复集合，成本相同按编号稳定选择，并检查总成本溢出。
    • Codec::assess_domains 按当前丢失状态评估单个机架或区域损失后的剩余分片和恢复缺口。

    要求 moonc >= 0.10.16。成本是宿主权重，故障域分析不推断概率，也不代替分片完整性校验；详见成本与故障域 API及区间更新。

    moon run examples/placement --target wasm-gc --deny-warn
moon run examples/range_update --target wasm-gc --deny-warn

    #Why this project

    MoonBit applications that distribute backups or packets need a reusable redundancy layer. A backup can tolerate loss of any m independent shards per stripe without retaining m complete replicas. An edge transfer can tolerate lost packets; a storage scrubber can identify damaged frames and regenerate only missing slots. The codec exposes multi-shard recovery, bounded object assembly and repair planning as reusable byte operations.

    #Install and run

    Install the MoonBit toolchain, then add the published library:

    moon add oyjh0381/moonerasure@0.3.0

    Import the library in your application's moon.pkg:

    import {
      "oyjh0381/moonerasure",
    }

    Source: oyjh0381/MoonErasure. Package: oyjh0381/moonerasure. Maintainer: oyjh0381.

    From the source repository, run:

    moon check --target all --deny-warn
moon build --target all --deny-warn
moon test --target all --deny-warn
moon run cmd/main --target wasm-gc

    The Mooncakes module name is oyjh0381/moonerasure. No third-party runtime packages are required. Development uses MoonBit compiler v0.10.16 or later. Version 0.1.0 does not include the selective-recovery or delta-update methods added in 0.2.0.

    #Core operations and explicit inputs

    OperationInputOutput / contract
    Codec::encode / encode_allk equal nonempty data shardsm parity shards / detached complete codeword
    Codec::verifyAll k+m shardsWhether parity agrees with data; no corruption localization
    Codec::reconstructk+m optional slots, at least k known-intactComplete codeword; surplus supplied shards cross-checked
    Codec::reconstruct_selectedOptional slots plus unique requested indicesOnly requested data/parity, in request order; surplus cross-checked
    Codec::update_parityOld parity and indexed old/new data pairsDetached new parity; unchanged data shards are not supplied
    Object / catalog methodsManifest and validated framesStripe/range recovery, repair frames, health and inventory

    // Missing slots remain None; only shard 1 is materialized in the result.
    let selected = try! codec.reconstruct_selected(surviving, [1])
    // Both old data and parity must belong to the same consistent generation.
    let changes : Array[@moonerasure.DataShardUpdate] = [
      { index: 3, old_data: old_block, new_data: new_block },
    ]
    let next_parity = try! codec.update_parity(old_parity, changes)

    For executable definitions of these variables and assertions, run the examples below. Persist new data and returned parity as one generation in the host storage adapter; this library does not provide durable commit semantics. An empty selected request only validates input shapes, returns [], and performs no codeword consistency check. A nonempty request requires at least k known-intact inputs, even when its requested shard is already present.

    #Minimal library use

    let codec = try! @moonerasure.Codec::new(3, 2)
    let object = try! @moonerasure.encode_object(
      codec,
      b"backup bytes",
      123,
      stripe_payload_limit=64,
    )
    let retained : Array[@moonerasure.ShardEnvelope] = []
    for frame in object.frames() {
      if frame.shard_index() != 1 && frame.shard_index() != 4 {
        retained.push(frame)
      }
    }
    let recovered = try! @moonerasure.recover_object(object.manifest(), retained)
    assert_eq(recovered.payload(), b"backup bytes")
    let replacements = try! @moonerasure.repair_object(object.manifest(), retained)

    The example drops two shards per stripe and reconstructs them from the remaining three. In production, store each frame's to_bytes() independently and store the 32-byte manifest separately with appropriate durability. Parse received bytes with ShardEnvelope::from_bytes; alternatively recover_serialized_object treats malformed or CRC-failed frames as known erasures and returns issue positions. Do not trust the shard index inside a damaged frame.

    For long-lived storage inventories, FrameCatalog::add_bytes validates CRC before admission, invalidate removes a suspect slot, and recover_range reads only relevant indexed stripes. See October features.

    #Runnable scenarios

    CommandDemonstrated need
    moon run cmd/mainBackup recovery and replacement frame generation after two placements disappear.
    moon run examples/transferOut-of-order packets, one absent data packet and one checksum-failed parity packet per stripe.
    moon run examples/scrubFrame inventory, health inspection, and persisted-slot repair.
    moon run examples/rangeRecover a byte range by fetching only overlapping stripes.
    moon run examples/streamEncode arbitrary input chunks and decode one stripe at a time.
    moon run examples/maintenanceAdmit a frame batch atomically and plan reads over currently recoverable byte ranges.
    moon run examples/degraded_readRead one missing 4 KiB data block from a 6+3 stripe without allocating unrelated missing shards.
    moon run examples/incremental_updateUpdate one 4 KiB block in a 10+4 stripe, compare parity with full encoding, then tolerate four erasures.
    moon run examples/selective_repairRebuild and replace only three failed backup placements, then verify the complete codeword.
    moon run examples/benchmark --target nativeVerified fixed workload for local timing of repeated loss patterns.

    All examples exit with a failing assertion if the round trip is wrong. API guide explains each public workflow; wire format records byte layouts and compatibility rules. See performance notes for complexity and reproducible timing guidance.

    #Capacity and failure boundary

    • 1 <= k,m <= 255, k + m <= 256; default encoded-byte budget is 16 MiB per stripe. The current implementation builds a dense generator matrix, so modest k and m are recommended in practice.
    • Default maximum object size is 64 MiB, default stripe payload limit 64 KiB, and maximum stripe count 4096. Callers can lower budgets; all public allocations are bounded. The in-memory complete-object API has a default 128 MiB encoded-output budget; the incremental encoder limits internal buffering to one stripe but returned frames can still fill memory if a caller holds them.
    • Recovery succeeds for erasures, where the failed shard is identified or rejected by CRC. More than m lost or rejected shards in one stripe is unrecoverable. CRC-32C is for accidental damage only; it is not encryption, authentication, a MAC, or a defense against maliciously crafted frames.
    • With exactly k surviving frames, Reed–Solomon parity provides no spare evidence to locate an unknown corrupted byte. Validate frame checksums and use authenticated storage or a cryptographic object digest when adversarial integrity matters.
    • Each object's manifest must be preserved. The library does not choose storage nodes, guarantee independent failure domains, transfer packets, perform I/O, or provide durable transactions.

    #Verification

    The repository includes black-box and white-box tests for GF arithmetic, matrix inversion, known vectors, exhaustive small erasure combinations, envelopes, bounds, corrupted bytes, multi-stripe objects, bundle parsing, incremental chunk boundaries, range reads, repair and health. CI checks/builds/tests all four backends, runs every example, and checks formatting and public interfaces. See testing notes.

    Pinned Go reference vectors cover encoding, full reconstruction, requested reconstruction and parity updates under the same GF(256) Vandermonde matrix. Ordinary moon test uses committed vectors and requires no Go installation. CI additionally regenerates them using the pinned Go module and checks for drift; the setup and reproduction commands are in the comparison document. This establishes agreement on the tested configurations, not universal interoperability with every matrix or Leopard mode.

    #Design and license

    See architecture, domain glossary, decision record and third-party notice. Licensed under Apache-2.0. No external source code or test corpus was copied into this repository.

    #十月第二轮：批量事务与可恢复区间

    FrameCatalog::add_many(frames, max_frames?) 在分离槽表上验证所有已解析帧，成功才提交；外部帧字节须先经 CRC 解析，批量操作不提供跨线程或持久化事务。recoverable_ranges(max_ranges?) 返回分片数量足够的最大连续半开字节区间，不解码整对象，也不认证分片内容。范围读取的参数是起点和长度：recover_range(range.start(), range.length())。

    运行 moon run examples/maintenance --target wasm-gc，展示两段可恢复数据与中间缺失条带。详见 功能与边界。

    #字节区间校验更新（0.3.0）

    update_parity_ranges 仅需变化区间的旧/新字节，支持同片不重叠多区间和严格预算；完整校验片仍会复制。

    运行 moon run examples/range_update --target wasm-gc，详见功能、预算与边界。本轮适配官方 MoonBit 0.10.16；受影响的 Windows 环境以 ./tools/windows/moon.ps1 替代 moon。以上接口从 0.3.0 起提供。

    ErasureError

    source
    pub(all) suberror ErasureError {
      InvalidConfiguration(String)
      InvalidShardCount(expected~ : Int, actual~ : Int)
      InvalidShardLength(index~ : Int, expected~ : Int, actual~ : Int)
      EmptyShard
      NotEnoughShards(required~ : Int, available~ : Int)
      SingularMatrix
      InvalidIndex(Int)
      DuplicateIndex(Int)
      InconsistentShard(Int)
      InvalidRange
      InvalidEnvelope(String)
      UnsupportedVersion(Int)
      ChecksumMismatch(Int)
      ResourceLimit(String)
      InvalidManifest(String)
    }

    Errors reported by configuration, coding, repair, or archive validation.

    ErasureError::code

    source
    fn ErasureError::code(self : ErasureError) -> String

    BundleStream

    source
    pub struct BundleStream {
      cursor : Int
      stage : Int
      expected_frames : Int
      parsed_frames : Int
      next_frame_length : Int
      manifest : Manifest?
      received_bytes : Int
      max_input_bytes : Int
      max_object_bytes : Int
      max_encoded_bytes : Int
      // private fields
    }

    Incremental parser for a complete bundle. Completed, checksum-validated frames are returned from push and are not retained by the parser.

    BundleStream::buffered_bytes

    source
    fn BundleStream::buffered_bytes(self : BundleStream) -> Int

    BundleStream::finish

    source
    fn BundleStream::finish(self : BundleStream) -> Manifest raise ErasureError

    Confirm that the full bundle ended exactly at a frame boundary.

    BundleStream::manifest

    source
    fn BundleStream::manifest(self : BundleStream) -> Manifest?

    BundleStream::new

    source
    fn BundleStream::new(max_input_bytes? : Int, max_object_bytes? : Int, max_encoded_bytes? : Int) -> BundleStream raise ErasureError

    BundleStream::parsed_frame_count

    source
    fn BundleStream::parsed_frame_count(self : BundleStream) -> Int

    BundleStream::push

    source
    fn BundleStream::push(self : BundleStream, chunk : Bytes) -> Array[ShardEnvelope] raise ErasureError

    Feed any nonempty or empty byte chunk. Return frames completed by it. Discard the stream after an error; earlier returned frames remain valid.

    Codec

    source
    pub struct Codec {
      data_count : Int
      parity_count : Int
      max_encoded_bytes : Int
      generator : Matrix
      field : Field
    }

    A systematic GF(256) codec. The first data_count shards are unchanged data; the following parity_count shards are computed redundancy.

    Codec::assess_domains

    source
    fn Codec::assess_domains(self : Codec, present : Array[Bool], placement : Array[String]) -> Array[DomainExposure] raise ErasureError

    Evaluate losing each named failure domain independently, given the current known-intact pattern. This does not certify placement independence or CRCs.

    Codec::data_count

    source
    fn Codec::data_count(self : Codec) -> Int

    Codec::encode

    source
    fn Codec::encode(self : Codec, data : Array[Bytes]) -> Array[Bytes] raise ErasureError

    Compute parity without changing the caller's data shards.

    Codec::encode_all

    source
    fn Codec::encode_all(self : Codec, data : Array[Bytes]) -> Array[Bytes] raise ErasureError

    Return detached data followed by parity shards.

    Codec::max_shard_bytes

    source
    fn Codec::max_shard_bytes(self : Codec) -> Int

    Codec::new

    source
    fn Codec::new(data_count : Int, parity_count : Int, max_encoded_bytes? : Int) -> Codec raise ErasureError

    Codec::parity_count

    source
    fn Codec::parity_count(self : Codec) -> Int

    Codec::plan

    source
    fn Codec::plan(self : Codec, present : Array[Bool]) -> RepairPlan raise ErasureError

    Report the first k present shards chosen by deterministic index order. Storage adapters can fetch additional shards before allocating decode work.

    Codec::plan_costed

    source
    fn Codec::plan_costed(self : Codec, present : Array[Bool], costs : Array[Int64]) -> CostedRepairPlan raise ErasureError

    Select at most k available shards by nonnegative host costs, tie-breaking by shard index. Costs are abstract units, not measured network latency.

    Codec::reconstruct

    source
    fn Codec::reconstruct(self : Codec, shards : Array[Bytes?]) -> Array[Bytes] raise ErasureError

    Reconstruct every erased shard from any data_count intact shards. Present shards are cross-checked against the reconstructed codeword. Exactly data_count present shards provide no spare evidence to detect an unknown corrupt byte; validate integrity envelopes before calling this API.

    Codec::reconstruct_selected

    source
    fn Codec::reconstruct_selected(self : Codec, shards : Array[Bytes?], requested : Array[Int]) -> Array[Bytes] raise ErasureError

    Return only the requested shards, detached and in request order. Both data and parity indices are supported. Unrequested missing shards are not built. For a nonempty request, at least k known-intact shards must be supplied. Extra supplied shards are checked for consistency with the selected k. An empty request validates shapes and indices but does no consistency work.

    Codec::total_count

    source
    fn Codec::total_count(self : Codec) -> Int

    Codec::update_parity

    source
    fn Codec::update_parity(self : Codec, parity : Array[Bytes], changes : Array[DataShardUpdate]) -> Array[Bytes] raise ErasureError

    Apply old XOR new deltas to parity, without reading unchanged data shards. Returns detached parity; neither inputs nor caller data slots are modified. All shapes and indices are validated before calculation. The caller must supply old data and parity from the same valid generation and persist the new data plus parity together. This operation cannot verify that premise.

    Codec::update_parity_ranges

    source
    fn Codec::update_parity_ranges(self : Codec, parity : Array[Bytes], changes : Array[DataRangeUpdate], max_ranges? : Int, max_changed_bytes? : Int) -> Array[Bytes] raise ErasureError

    Update parity from only modified data ranges. Ranges in the same data shard must not overlap; input order does not matter. Parity is copied in full, so output cost remains O(m*L). All inputs must share one old generation.

    Codec::verify

    source
    fn Codec::verify(self : Codec, shards : Array[Bytes]) -> Bool raise ErasureError

    Check whether every supplied parity shard agrees with the data shards.

    CostedRepairPlan

    source
    pub struct CostedRepairPlan {
      total_cost : Int64
      // private fields
    }

    CostedRepairPlan::plan

    source

    CostedRepairPlan::total_cost

    source
    fn CostedRepairPlan::total_cost(self : CostedRepairPlan) -> Int64

    DataRangeUpdate

    source
    pub(all) struct DataRangeUpdate {
      index : Int
      offset : Int
      old_data : Bytes
      new_data : Bytes
    }

    Nonempty, same-length old/new byte range within one data shard.

    DataShardUpdate

    source
    pub(all) struct DataShardUpdate {
      index : Int
      old_data : Bytes
      new_data : Bytes
    }

    A same-size replacement of one data shard in a consistent old codeword.

    DecoderEntry

    source
    type DecoderEntry

    DomainExposure

    source
    pub struct DomainExposure {
      domain : String
      lost_available : Int
      remaining : Int
      spare : Int
    }

    DomainExposure::domain

    source
    fn DomainExposure::domain(self : DomainExposure) -> String

    DomainExposure::lost_available

    source
    fn DomainExposure::lost_available(self : DomainExposure) -> Int

    DomainExposure::recoverable

    source
    fn DomainExposure::recoverable(self : DomainExposure) -> Bool

    DomainExposure::remaining

    source
    fn DomainExposure::remaining(self : DomainExposure) -> Int

    DomainExposure::spare

    source
    fn DomainExposure::spare(self : DomainExposure) -> Int

    Remaining known-intact shards minus k. Negative is a recovery deficit.

    EncodedObject

    source
    pub struct EncodedObject {
      manifest : Manifest
      // private fields
    }

    A bounded in-memory object ready for caller-controlled placement.

    EncodedObject::frame_count

    source
    fn EncodedObject::frame_count(self : EncodedObject) -> Int

    EncodedObject::frames

    source

    EncodedObject::manifest

    source
    fn EncodedObject::manifest(self : EncodedObject) -> Manifest

    Field

    source
    pub struct Field {
      // private fields
    }

    GF(2^8) with irreducible polynomial 0x11d and generator 2. Keeping tables inside a codec avoids mutable global state across backends.

    FrameCatalog

    source
    pub struct FrameCatalog {
      manifest : Manifest
      stored_count : Int
      // private fields
    }

    In-memory index for independently stored frames of one manifest. A storage adapter may use it to decide which stripe to fetch or repair.

    FrameCatalog::add

    source
    fn FrameCatalog::add(self : FrameCatalog, frame : ShardEnvelope) -> Unit raise ErasureError

    Register one already parsed frame. Identity, dimensions, original stripe length and duplicate index are checked before the catalog changes.

    FrameCatalog::add_bytes

    source
    fn FrameCatalog::add_bytes(self : FrameCatalog, wire : Bytes, max_encoded_bytes? : Int) -> Unit raise ErasureError

    Parse a serialized frame and admit it only if both its CRC and catalog identity checks pass. A failed call leaves the catalog unchanged.

    FrameCatalog::add_many

    source
    fn FrameCatalog::add_many(self : FrameCatalog, frames : Array[ShardEnvelope], max_frames? : Int) -> Int raise ErasureError

    Admit a batch atomically. Validate on a detached slot table, then publish only after every identity and duplicate check succeeds. Frames are already parsed; use add_bytes for CRC validation of external serialized frames.

    FrameCatalog::capacity

    source
    fn FrameCatalog::capacity(self : FrameCatalog) -> Int

    FrameCatalog::get

    source
    fn FrameCatalog::get(self : FrameCatalog, stripe_index : Int, shard_index : Int) -> ShardEnvelope? raise ErasureError

    Look up one slot. A missing slot is returned as None.

    FrameCatalog::invalidate

    source
    fn FrameCatalog::invalidate(self : FrameCatalog, stripe_index : Int, shard_index : Int) -> Bool raise ErasureError

    Evict a slot after a storage read or checksum check shows that its bytes can no longer be trusted. Removing an absent slot is a harmless no-op.

    FrameCatalog::manifest

    source
    fn FrameCatalog::manifest(self : FrameCatalog) -> Manifest

    FrameCatalog::missing_indices

    source
    fn FrameCatalog::missing_indices(self : FrameCatalog, stripe_index : Int) -> Array[Int] raise ErasureError

    Return the missing shard indices of a stripe in ascending order.

    FrameCatalog::new

    source
    fn FrameCatalog::new(manifest : Manifest) -> FrameCatalog

    FrameCatalog::recover_range

    source
    fn FrameCatalog::recover_range(self : FrameCatalog, start : Int, length : Int, max_encoded_bytes? : Int) -> Bytes raise ErasureError

    Decode only the stripes intersecting a requested range from indexed frames. This is useful after an application has inventoried unordered stored fragments but needs a small portion of the original object.

    FrameCatalog::recover_stripe

    source
    fn FrameCatalog::recover_stripe(self : FrameCatalog, stripe_index : Int, max_encoded_bytes? : Int) -> StripeRecovery raise ErasureError

    Decode an indexed stripe. Present bytes are still cross-checked by recover_stripe, since catalog admission validates metadata only.

    FrameCatalog::recoverable_ranges

    source
    fn FrameCatalog::recoverable_ranges(self : FrameCatalog, max_ranges? : Int) -> Array[RecoverableRange] raise ErasureError

    Enumerate maximal contiguous ranges whose stripes each have at least k admitted frames. Supports partial media reads and resume scheduling without decoding the object. A result budget is checked before every emitted run.

    FrameCatalog::stored_count

    source
    fn FrameCatalog::stored_count(self : FrameCatalog) -> Int

    FrameCatalog::stripe_frames

    source
    fn FrameCatalog::stripe_frames(self : FrameCatalog, stripe_index : Int) -> Array[ShardEnvelope] raise ErasureError

    Return present frames of a stripe in canonical shard index order.

    FrameCatalog::stripe_recoverable

    source
    fn FrameCatalog::stripe_recoverable(self : FrameCatalog, stripe_index : Int) -> Bool raise ErasureError

    FrameIssue

    source
    pub struct FrameIssue {
      source_position : Int
      code : String
    }

    A rejected input frame is identified by its position in the input list. The shard index inside a damaged header is deliberately not trusted.

    FrameIssue::code

    source
    fn FrameIssue::code(self : FrameIssue) -> String

    FrameIssue::source_position

    source
    fn FrameIssue::source_position(self : FrameIssue) -> Int

    FrameScan

    source
    pub struct FrameScan {
      // private fields
    }

    FrameScan::issues

    source
    fn FrameScan::issues(self : FrameScan) -> Array[FrameIssue]

    FrameScan::rejected_count

    source
    fn FrameScan::rejected_count(self : FrameScan) -> Int

    FrameScan::valid

    source
    fn FrameScan::valid(self : FrameScan) -> Array[ShardEnvelope]

    FrameScan::valid_count

    source
    fn FrameScan::valid_count(self : FrameScan) -> Int

    Manifest

    source
    pub struct Manifest {
      set_id : Int
      data_count : Int
      parity_count : Int
      stripe_payload_limit : Int
      stripe_count : Int
      total_length : Int
    }

    Bounded description of an ordered sequence of independently coded stripes.

    Manifest::data_count

    source
    fn Manifest::data_count(self : Manifest) -> Int

    Manifest::from_bytes

    source
    fn Manifest::from_bytes(input : Bytes, max_object_bytes? : Int, max_encoded_bytes? : Int) -> Manifest raise ErasureError

    Manifest::new

    source
    fn Manifest::new(codec : Codec, set_id : Int, stripe_payload_limit : Int, total_length : Int, max_object_bytes? : Int) -> Manifest raise ErasureError

    Manifest::parity_count

    source
    fn Manifest::parity_count(self : Manifest) -> Int

    Manifest::set_id

    source
    fn Manifest::set_id(self : Manifest) -> Int

    Manifest::stripe_count

    source
    fn Manifest::stripe_count(self : Manifest) -> Int

    Manifest::stripe_length

    source
    fn Manifest::stripe_length(self : Manifest, index : Int) -> Int raise ErasureError

    Manifest::stripe_payload_limit

    source
    fn Manifest::stripe_payload_limit(self : Manifest) -> Int

    Manifest::to_bytes

    source
    fn Manifest::to_bytes(self : Manifest) -> Bytes

    28 bytes of metadata followed by CRC32C of those 28 bytes.

    Manifest::total_length

    source
    fn Manifest::total_length(self : Manifest) -> Int

    Matrix

    source
    pub struct Matrix {
      rows : Int
      cols : Int
      // private fields
    }

    Small field matrix used only for codec construction and erasure recovery.

    ObjectDecoder

    source
    pub struct ObjectDecoder {
      codec : Codec
      manifest : Manifest
      next_stripe : Int
      decoded_bytes : Int
      repaired_shards : Int
    }

    Sequential decoder for objects whose frames are fetched stripe by stripe. The caller groups each stripe's available frames and can write each returned payload chunk directly to a destination.

    ObjectDecoder::decoded_bytes

    source
    fn ObjectDecoder::decoded_bytes(self : ObjectDecoder) -> Int

    ObjectDecoder::finish

    source
    fn ObjectDecoder::finish(self : ObjectDecoder) -> Int raise ErasureError

    Confirm that every stripe was consumed and the advertised byte count matches. There is no hidden payload buffer in this decoder.

    ObjectDecoder::new

    source
    fn ObjectDecoder::new(manifest : Manifest, max_encoded_bytes? : Int) -> ObjectDecoder raise ErasureError

    ObjectDecoder::next_stripe

    source
    fn ObjectDecoder::next_stripe(self : ObjectDecoder) -> Int

    ObjectDecoder::push_stripe

    source
    fn ObjectDecoder::push_stripe(self : ObjectDecoder, frames : Array[ShardEnvelope]) -> Bytes raise ErasureError

    Decode exactly the next manifest stripe. On failure, the cursor remains unchanged so callers may fetch an alternative frame and retry.

    ObjectDecoder::repaired_shards

    source
    fn ObjectDecoder::repaired_shards(self : ObjectDecoder) -> Int

    ObjectEncoder

    source
    pub struct ObjectEncoder {
      codec : Codec
      manifest : Manifest
      accepted_bytes : Int
      emitted_stripes : Int
      // private fields
    }

    Incremental object encoder with an upfront manifest. Applications can persist completed frames immediately, keeping only one stripe in memory.

    ObjectEncoder::accepted_bytes

    source
    fn ObjectEncoder::accepted_bytes(self : ObjectEncoder) -> Int

    ObjectEncoder::buffered_bytes

    source
    fn ObjectEncoder::buffered_bytes(self : ObjectEncoder) -> Int

    ObjectEncoder::emitted_stripes

    source
    fn ObjectEncoder::emitted_stripes(self : ObjectEncoder) -> Int

    ObjectEncoder::finish

    source
    fn ObjectEncoder::finish(self : ObjectEncoder) -> Manifest raise ErasureError

    Confirm the stream ended exactly at the declared length and every stripe was emitted. Empty objects finish without emitting frames.

    ObjectEncoder::manifest

    source
    fn ObjectEncoder::manifest(self : ObjectEncoder) -> Manifest

    ObjectEncoder::new

    source
    fn ObjectEncoder::new(codec : Codec, set_id : Int, total_length : Int, stripe_payload_limit? : Int, max_object_bytes? : Int) -> ObjectEncoder raise ErasureError

    ObjectEncoder::push

    source
    fn ObjectEncoder::push(self : ObjectEncoder, chunk : Bytes) -> Array[ShardEnvelope] raise ErasureError

    Accept a chunk of the declared object. A chunk may complete zero, one, or many stripes. Exceeding the advertised size is rejected before mutation.

    ObjectHealth

    source
    pub struct ObjectHealth {
      healthy_count : Int
      repairable_count : Int
      unrecoverable_count : Int
      invalid_count : Int
      // private fields
    }

    ObjectHealth::fully_recoverable

    source
    fn ObjectHealth::fully_recoverable(self : ObjectHealth) -> Bool

    ObjectHealth::healthy_count

    source
    fn ObjectHealth::healthy_count(self : ObjectHealth) -> Int

    ObjectHealth::invalid_count

    source
    fn ObjectHealth::invalid_count(self : ObjectHealth) -> Int

    ObjectHealth::repairable_count

    source
    fn ObjectHealth::repairable_count(self : ObjectHealth) -> Int

    ObjectHealth::stripes

    source

    ObjectHealth::unrecoverable_count

    source
    fn ObjectHealth::unrecoverable_count(self : ObjectHealth) -> Int

    ObjectRecovery

    source
    pub struct ObjectRecovery {
      payload : Bytes
      repaired_shard_count : Int
      stripe_count : Int
    }

    ObjectRecovery::payload

    source
    fn ObjectRecovery::payload(self : ObjectRecovery) -> Bytes

    ObjectRecovery::repaired_shard_count

    source
    fn ObjectRecovery::repaired_shard_count(self : ObjectRecovery) -> Int

    ObjectRecovery::stripe_count

    source
    fn ObjectRecovery::stripe_count(self : ObjectRecovery) -> Int

    ObjectRepair

    source
    pub struct ObjectRepair {
      repaired_stripes : Int
      // private fields
    }

    Replacement frames for missing slots, in stripe then shard order. The caller decides where to persist them. Existing frames are never rewritten.

    ObjectRepair::repaired_stripes

    source
    fn ObjectRepair::repaired_stripes(self : ObjectRepair) -> Int

    ObjectRepair::replacement_count

    source
    fn ObjectRepair::replacement_count(self : ObjectRepair) -> Int

    ObjectRepair::replacements

    source
    fn ObjectRepair::replacements(self : ObjectRepair) -> Array[ShardEnvelope]

    RecoverableRange

    source
    pub struct RecoverableRange {
      start : Int
      end : Int
    }

    A half-open byte range that has enough admitted frames to attempt recovery. Presence is not authentication or a new content checksum verification.

    RecoverableRange::end

    source
    fn RecoverableRange::end(self : RecoverableRange) -> Int

    RecoverableRange::length

    source
    fn RecoverableRange::length(self : RecoverableRange) -> Int

    RecoverableRange::start

    source
    fn RecoverableRange::start(self : RecoverableRange) -> Int

    RecoverySession

    source
    pub struct RecoverySession {
      codec : Codec
      capacity : Int
      next_evict : Int
      hits : Int
      misses : Int
      // private fields
    }

    Reuse decode matrices when many stripes have the same erasure pattern. Each session owns a small bounded FIFO cache; separate sessions should be used for concurrent callers. Cached matrices contain no payload bytes.

    RecoverySession::cache_hits

    source
    fn RecoverySession::cache_hits(self : RecoverySession) -> Int

    RecoverySession::cache_misses

    source
    fn RecoverySession::cache_misses(self : RecoverySession) -> Int

    RecoverySession::cache_size

    source
    fn RecoverySession::cache_size(self : RecoverySession) -> Int

    RecoverySession::new

    source
    fn RecoverySession::new(codec : Codec, max_patterns? : Int) -> RecoverySession raise ErasureError

    RecoverySession::reconstruct

    source
    fn RecoverySession::reconstruct(self : RecoverySession, shards : Array[Bytes?]) -> Array[Bytes] raise ErasureError

    Reconstruct with the same validation as Codec::reconstruct. Data-only stripes skip matrix work and do not alter cache counters. Cache misses calculate a dense inverse once and reuse it on subsequent matching stripes.

    RepairPlan

    source
    pub struct RepairPlan {
      required : Int
      available : Int
      // private fields
    }

    A deterministic preflight for a pattern of present and erased shards. It does not read shard bytes or make an integrity assertion.

    RepairPlan::available

    source
    fn RepairPlan::available(self : RepairPlan) -> Int

    RepairPlan::minimum_additional_shards

    source
    fn RepairPlan::minimum_additional_shards(self : RepairPlan) -> Int

    RepairPlan::missing_data_indices

    source
    fn RepairPlan::missing_data_indices(self : RepairPlan) -> Array[Int]

    RepairPlan::missing_parity_indices

    source
    fn RepairPlan::missing_parity_indices(self : RepairPlan) -> Array[Int]

    RepairPlan::needs_decode

    source
    fn RepairPlan::needs_decode(self : RepairPlan) -> Bool

    RepairPlan::recoverable

    source
    fn RepairPlan::recoverable(self : RepairPlan) -> Bool

    RepairPlan::required

    source
    fn RepairPlan::required(self : RepairPlan) -> Int

    RepairPlan::selected_indices

    source
    fn RepairPlan::selected_indices(self : RepairPlan) -> Array[Int]

    ScannedObjectHealth

    source
    pub struct ScannedObjectHealth {
      health : ObjectHealth
      // private fields
    }

    Health inspection of raw frames: invalid encodings are reported by input position, while stripe status is computed from surviving valid frames.

    ScannedObjectHealth::health

    source

    ScannedObjectHealth::issues

    source

    ScannedObjectRecovery

    source
    pub struct ScannedObjectRecovery {
      recovery : ObjectRecovery
      // private fields
    }

    Object recovery together with diagnostics for rejected raw frame positions.

    ScannedObjectRecovery::issues

    source

    ScannedObjectRecovery::recovery

    source

    ScannedObjectRepair

    source
    pub struct ScannedObjectRepair {
      repair : ObjectRepair
      // private fields
    }

    Repair result for raw frame inputs. Damaged frames are reported by input position and treated as absent; replacement frames are checksum-protected.

    ScannedObjectRepair::issues

    source

    ScannedObjectRepair::repair

    source

    ScannedRangeRecovery

    source
    pub struct ScannedRangeRecovery {
      payload : Bytes
      // private fields
    }

    A recovered byte range and diagnostics for raw frames rejected by CRC or structural validation.

    ScannedRangeRecovery::issues

    source

    ScannedRangeRecovery::payload

    source
    fn ScannedRangeRecovery::payload(self : ScannedRangeRecovery) -> Bytes

    ScannedRecovery

    source
    pub struct ScannedRecovery {
      recovery : StripeRecovery
      // private fields
    }

    ScannedRecovery::issues

    source

    ScannedRecovery::recovery

    source

    ShardEnvelope

    source
    pub struct ShardEnvelope {
      set_id : Int
      stripe_index : Int
      shard_index : Int
      data_count : Int
      parity_count : Int
      original_length : Int
      payload : Bytes
    }

    Version 1 frame: 28-byte metadata, 4-byte CRC32C, then shard payload. Integer fields are unsigned little-endian; ids and lengths are capped at signed 31-bit values so they behave identically on all MoonBit targets.

    ShardEnvelope::data_count

    source
    fn ShardEnvelope::data_count(self : ShardEnvelope) -> Int

    ShardEnvelope::from_bytes

    source
    fn ShardEnvelope::from_bytes(input : Bytes, max_encoded_bytes? : Int) -> ShardEnvelope raise ErasureError

    Parse one bounded frame. Invalid metadata and damaged payloads fail closed.

    ShardEnvelope::new

    source
    fn ShardEnvelope::new(codec : Codec, set_id : Int, stripe_index : Int, shard_index : Int, original_length : Int, payload : Bytes) -> ShardEnvelope raise ErasureError

    ShardEnvelope::original_length

    source
    fn ShardEnvelope::original_length(self : ShardEnvelope) -> Int

    ShardEnvelope::parity_count

    source
    fn ShardEnvelope::parity_count(self : ShardEnvelope) -> Int

    ShardEnvelope::payload

    source
    fn ShardEnvelope::payload(self : ShardEnvelope) -> Bytes

    ShardEnvelope::set_id

    source
    fn ShardEnvelope::set_id(self : ShardEnvelope) -> Int

    ShardEnvelope::shard_index

    source
    fn ShardEnvelope::shard_index(self : ShardEnvelope) -> Int

    ShardEnvelope::stripe_index

    source
    fn ShardEnvelope::stripe_index(self : ShardEnvelope) -> Int

    ShardEnvelope::to_bytes

    source
    fn ShardEnvelope::to_bytes(self : ShardEnvelope) -> Bytes

    Serialize a validated envelope with a CRC covering metadata and payload.

    StripeHealth

    source
    pub struct StripeHealth {
      stripe_index : Int
      status : StripeStatus
      present_count : Int
      reason_code : String?
      // private fields
    }

    StripeHealth::missing_indices

    source
    fn StripeHealth::missing_indices(self : StripeHealth) -> Array[Int]

    StripeHealth::present_count

    source
    fn StripeHealth::present_count(self : StripeHealth) -> Int

    StripeHealth::reason_code

    source
    fn StripeHealth::reason_code(self : StripeHealth) -> String?

    StripeHealth::status

    source
    fn StripeHealth::status(self : StripeHealth) -> StripeStatus

    StripeHealth::stripe_index

    source
    fn StripeHealth::stripe_index(self : StripeHealth) -> Int

    StripeRecovery

    source
    pub struct StripeRecovery {
      payload : Bytes
      provided_count : Int
      // private fields
    }

    A complete repaired stripe plus information about which shards were absent.

    StripeRecovery::missing_indices

    source
    fn StripeRecovery::missing_indices(self : StripeRecovery) -> Array[Int]

    StripeRecovery::payload

    source
    fn StripeRecovery::payload(self : StripeRecovery) -> Bytes

    StripeRecovery::provided_count

    source
    fn StripeRecovery::provided_count(self : StripeRecovery) -> Int

    StripeRecovery::shards

    source

    StripeStatus

    source
    pub(all) enum StripeStatus {
      Healthy
      Repairable
      Unrecoverable
      InvalidInput
    }

    Result of examining one stripe without returning payload bytes.

    StripeStatus::label

    source
    fn StripeStatus::label(self : StripeStatus) -> String

    crc32c

    source
    fn crc32c(input : Bytes) -> UInt

    encode_object

    source
    fn encode_object(codec : Codec, payload : Bytes, set_id : Int, stripe_payload_limit? : Int, max_object_bytes? : Int, max_output_bytes? : Int) -> EncodedObject raise ErasureError

    Split an object into bounded, independently recoverable stripes.

    encode_stripe

    source
    fn encode_stripe(codec : Codec, payload : Bytes, set_id : Int, stripe_index : Int) -> Array[ShardEnvelope] raise ErasureError

    Split one nonempty payload into equal padded data shards and encode parity.

    export_bundle

    source
    fn export_bundle(object : EncodedObject, max_output_bytes? : Int) -> Bytes raise ErasureError

    Single-file archive: 24-byte bundle header, 32-byte manifest, then a length-prefixed sequence of complete shard frames. Individual frames retain their own checksums so they can also be extracted and stored independently.

    import_bundle

    source
    fn import_bundle(input : Bytes, max_input_bytes? : Int, max_object_bytes? : Int, max_encoded_bytes? : Int) -> EncodedObject raise ErasureError

    Strictly import one complete bundle and validate every stripe. This archive path refuses damage; individual frame recovery remains available through recover_serialized_stripe when some bytes are unavailable.

    inspect_object

    source
    fn inspect_object(manifest : Manifest, frames : Array[ShardEnvelope], max_encoded_bytes? : Int) -> ObjectHealth raise ErasureError

    Inspect every stripe against a manifest. The caller may decide which repairable stripes to reconstruct and where to place replacement shards.

    inspect_serialized_object

    source
    fn inspect_serialized_object(manifest : Manifest, inputs : Array[Bytes], max_encoded_bytes? : Int, max_total_bytes? : Int) -> ScannedObjectHealth raise ErasureError

    Inspect an object directly from independently stored frame bytes. A bad checksum is counted as missing capacity, but a valid frame with conflicting set id, dimensions or duplicate slot marks input invalid.

    inspect_stripe

    source
    fn inspect_stripe(codec : Codec, frames : Array[ShardEnvelope], set_id : Int, stripe_index : Int) -> StripeHealth

    Plan an individual stripe and verify consistency when enough frames exist.

    recover_object

    source
    fn recover_object(manifest : Manifest, frames : Array[ShardEnvelope], max_encoded_bytes? : Int) -> ObjectRecovery raise ErasureError

    Reassemble in manifest order; every stripe must have enough intact frames. For individual repaired shard bytes, call recover_stripe on that stripe.

    recover_object_range

    source
    fn recover_object_range(manifest : Manifest, frames : Array[ShardEnvelope], start : Int, length : Int, max_encoded_bytes? : Int) -> Bytes raise ErasureError

    Recover only the stripes overlapping a byte range. Callers may supply frames for the whole object or only for the requested stripes. Damage in unrelated stripes does not block a range read.

    recover_serialized_object

    source
    fn recover_serialized_object(manifest : Manifest, inputs : Array[Bytes], max_encoded_bytes? : Int, max_total_bytes? : Int) -> ScannedObjectRecovery raise ErasureError

    Recover an object from independently received shard frames. Corrupt or malformed frames become known erasures; a mismatch among otherwise valid frames is still rejected by stripe reconstruction.

    recover_serialized_range

    source
    fn recover_serialized_range(manifest : Manifest, inputs : Array[Bytes], start : Int, length : Int, max_encoded_bytes? : Int, max_total_bytes? : Int) -> ScannedRangeRecovery raise ErasureError

    Recover an object byte range from independently stored serialized frames. Only intersecting stripes are decoded. Damaged frames within those stripes are erasures and must fit the parity budget.

    recover_serialized_stripe

    source
    fn recover_serialized_stripe(codec : Codec, inputs : Array[Bytes], set_id : Int, stripe_index : Int) -> ScannedRecovery raise ErasureError

    Treat checksum-failed or malformed frames as erasures, while retaining a diagnostic for each rejected input position. A sufficient set of valid frames is still required.

    recover_stripe

    source
    fn recover_stripe(codec : Codec, frames : Array[ShardEnvelope], set_id : Int, stripe_index : Int) -> StripeRecovery raise ErasureError

    Recover a stripe from validated envelopes. Duplicate indices, mixed identities, mismatched sizes and inconsistent extra shards are rejected.

    repair_object

    source
    fn repair_object(manifest : Manifest, frames : Array[ShardEnvelope], max_encoded_bytes? : Int) -> ObjectRepair raise ErasureError

    Regenerate only absent frames. Every supplied frame is checked by the stripe decoder; missing frames beyond parity capacity abort the operation. This operation is all-or-error: callers receive no partial repair list.

    repair_serialized_object

    source
    fn repair_serialized_object(manifest : Manifest, inputs : Array[Bytes], max_encoded_bytes? : Int, max_total_bytes? : Int) -> ScannedObjectRepair raise ErasureError

    Scan independent serialized frames and regenerate every absent or damaged slot. The returned issue positions let storage adapters quarantine bad objects without trusting their internal shard indices.

    scan_frames

    source
    fn scan_frames(inputs : Array[Bytes], max_frames? : Int, max_encoded_bytes? : Int, max_total_bytes? : Int) -> FrameScan raise ErasureError

    Parse independent frames without discarding good neighbors. The caller must still validate stripe identity and duplicates when using the valid frames.

    Source Files

    Powered by MoonBit

    Site sourceReport issueModulesSkillsCLIBuild queueStatistics

    © 2026 mooncakes.io