🥮 mooncakes.io

    moonerasure

    Portable Reed-Solomon shard recovery and integrity envelopes

    erasure
    reed-solomon
    storage
    recovery
    gf256
    Download zip
    Author
    Version
    0.1.0
    License
    Apache-2.0
    Repository
    Last updated
    14 hours ago
    Downloads
    2

    #MoonErasure

    MoonErasure is a MoonBit library for systematic Reed–Solomon erasure coding over GF(256). It turns k data shards into k + m shards and reconstructs a stripe from any k intact shards. The first k shards retain the original padded data. Applications decide where to store, send, authenticate and persist the shards.

    The library supports complete objects, bounded stripes, independent checksum-protected frames, repair plans, range reads, incremental encoding and decoding, a portable single-file bundle format, and health inspection. The code is original MoonBit and runs on wasm, wasm-gc, js and native targets.

    #Why this project

    MoonBit applications that distribute backups or packets need a reusable redundancy layer. A backup can tolerate loss of any m independent shards per stripe without retaining m complete replicas. An edge transfer can tolerate lost packets; a storage scrubber can identify damaged frames and regenerate only missing slots. Topic research explains the nearest mooncakes.io project found and the distinct contribution. Search results are time dependent and must be repeated before release.

    #Install and run

    Install the MoonBit toolchain, then add the published library:

    moon add oyjh0381/moonerasure@0.1.0

    Import the library in your application's moon.pkg:

    import {
      "oyjh0381/moonerasure",
    }

    Source: oyjh0381/MoonErasure. Package: oyjh0381/moonerasure. Maintainer: oyjh0381.

    From the source repository, run:

    moon check --target all --deny-warn
moon build --target all --deny-warn
moon test --target all --deny-warn
moon run cmd/main --target wasm-gc

    The Mooncakes module name is oyjh0381/moonerasure. No third-party runtime packages are required. Release and verification evidence is recorded in the 0.1.0 release record.

    #Minimal library use

    let codec = try! @moonerasure.Codec::new(3, 2)
    let object = try! @moonerasure.encode_object(
      codec,
      b"backup bytes",
      123,
      stripe_payload_limit=64,
    )
    let retained : Array[@moonerasure.ShardEnvelope] = []
    for frame in object.frames() {
      if frame.shard_index() != 1 && frame.shard_index() != 4 {
        retained.push(frame)
      }
    }
    let recovered = try! @moonerasure.recover_object(object.manifest(), retained)
    assert_eq(recovered.payload(), b"backup bytes")
    let replacements = try! @moonerasure.repair_object(object.manifest(), retained)

    The example drops two shards per stripe and reconstructs them from the remaining three. In production, store each frame's to_bytes() independently and store the 32-byte manifest separately with appropriate durability. Parse received bytes with ShardEnvelope::from_bytes; alternatively recover_serialized_object treats malformed or CRC-failed frames as known erasures and returns issue positions. Do not trust the shard index inside a damaged frame.

    For long-lived storage inventories, FrameCatalog::add_bytes validates CRC before admission, invalidate removes a suspect slot, and recover_range reads only relevant indexed stripes. See October features.

    #Runnable scenarios

    CommandDemonstrated need
    moon run cmd/mainBackup recovery and replacement frame generation after two placements disappear.
    moon run examples/transferOut-of-order packets, one absent data packet and one checksum-failed parity packet per stripe.
    moon run examples/scrubFrame inventory, health inspection, and persisted-slot repair.
    moon run examples/rangeRecover a byte range by fetching only overlapping stripes.
    moon run examples/streamEncode arbitrary input chunks and decode one stripe at a time.
    moon run examples/maintenanceAdmit a frame batch atomically and plan reads over currently recoverable byte ranges.
    moon run examples/benchmark --target nativeVerified fixed workload for local timing of repeated loss patterns.

    All examples exit with a failing assertion if the round trip is wrong. API guide explains each public workflow; wire format records byte layouts and compatibility rules. See performance notes for complexity and reproducible timing guidance.

    #Capacity and failure boundary

    • 1 <= k,m <= 255, k + m <= 256; default encoded-byte budget is 16 MiB per stripe. The current implementation builds a dense generator matrix, so modest k and m are recommended in practice.
    • Default maximum object size is 64 MiB, default stripe payload limit 64 KiB, and maximum stripe count 4096. Callers can lower budgets; all public allocations are bounded. The in-memory complete-object API has a default 128 MiB encoded-output budget; the incremental encoder limits internal buffering to one stripe but returned frames can still fill memory if a caller holds them.
    • Recovery succeeds for erasures, where the failed shard is identified or rejected by CRC. More than m lost or rejected shards in one stripe is unrecoverable. CRC-32C is for accidental damage only; it is not encryption, authentication, a MAC, or a defense against maliciously crafted frames.
    • With exactly k surviving frames, Reed–Solomon parity provides no spare evidence to locate an unknown corrupted byte. Validate frame checksums and use authenticated storage or a cryptographic object digest when adversarial integrity matters.
    • Each object's manifest must be preserved. The library does not choose storage nodes, guarantee independent failure domains, transfer packets, perform I/O, or provide durable transactions.

    #Verification

    The repository includes black-box and white-box tests for GF arithmetic, matrix inversion, known vectors, exhaustive small erasure combinations, envelopes, bounds, corrupted bytes, multi-stripe objects, bundle parsing, incremental chunk boundaries, range reads, repair and health. CI checks/builds/tests all four backends, runs every example, and checks formatting and public interfaces. See testing notes.

    #Design and license

    See architecture, domain glossary, decision record, selection research, local review and third-party notice. Licensed under Apache-2.0. No external source code or test corpus was copied into this repository.

    #十月第二轮：批量事务与可恢复区间

    FrameCatalog::add_many(frames, max_frames?) 在分离槽表上验证所有已解析帧，成功才提交；外部帧字节须先经 CRC 解析，批量操作不提供跨线程或持久化事务。recoverable_ranges(max_ranges?) 返回分片数量足够的最大连续半开字节区间，不解码整对象，也不认证分片内容。范围读取的参数是起点和长度：recover_range(range.start(), range.length())。

    运行 moon run examples/maintenance --target wasm-gc，展示两段可恢复数据与中间缺失条带。详见 本轮审查与复杂度 和 十月申报资料稿。

    ErasureError

    source
    pub(all) suberror ErasureError {
      InvalidConfiguration(String)
      InvalidShardCount(expected~ : Int, actual~ : Int)
      InvalidShardLength(index~ : Int, expected~ : Int, actual~ : Int)
      EmptyShard
      NotEnoughShards(required~ : Int, available~ : Int)
      SingularMatrix
      InvalidIndex(Int)
      DuplicateIndex(Int)
      InconsistentShard(Int)
      InvalidRange
      InvalidEnvelope(String)
      UnsupportedVersion(Int)
      ChecksumMismatch(Int)
      ResourceLimit(String)
      InvalidManifest(String)
    }

    Errors reported by configuration, coding, repair, or archive validation.

    ErasureError::code

    source
    fn ErasureError::code(self : ErasureError) -> String

    BundleStream

    source
    pub struct BundleStream {
      pending : Array[Byte]
      cursor : Int
      stage : Int
      expected_frames : Int
      parsed_frames : Int
      next_frame_length : Int
      manifest : Manifest?
      seen : Array[Bool]
      received_bytes : Int
      max_input_bytes : Int
      max_object_bytes : Int
      max_encoded_bytes : Int
    }

    Incremental parser for a complete bundle. Completed, checksum-validated frames are returned from push and are not retained by the parser.

    BundleStream::buffered_bytes

    source
    fn BundleStream::buffered_bytes(self : BundleStream) -> Int

    BundleStream::finish

    source
    fn BundleStream::finish(self : BundleStream) -> Manifest raise ErasureError

    Confirm that the full bundle ended exactly at a frame boundary.

    BundleStream::manifest

    source
    fn BundleStream::manifest(self : BundleStream) -> Manifest?

    BundleStream::new

    source
    fn BundleStream::new(max_input_bytes? : Int, max_object_bytes? : Int, max_encoded_bytes? : Int) -> BundleStream raise ErasureError

    BundleStream::parsed_frame_count

    source
    fn BundleStream::parsed_frame_count(self : BundleStream) -> Int

    BundleStream::push

    source
    fn BundleStream::push(self : BundleStream, chunk : Bytes) -> Array[ShardEnvelope] raise ErasureError

    Feed any nonempty or empty byte chunk. Return frames completed by it. Discard the stream after an error; earlier returned frames remain valid.

    Codec

    source
    pub struct Codec {
      data_count : Int
      parity_count : Int
      max_encoded_bytes : Int
      generator : Matrix
      field : Field
    }

    A systematic GF(256) codec. The first data_count shards are unchanged data; the following parity_count shards are computed redundancy.

    Codec::data_count

    source
    fn Codec::data_count(self : Codec) -> Int

    Codec::encode

    source
    fn Codec::encode(self : Codec, data : Array[Bytes]) -> Array[Bytes] raise ErasureError

    Compute parity without changing the caller's data shards.

    Codec::encode_all

    source
    fn Codec::encode_all(self : Codec, data : Array[Bytes]) -> Array[Bytes] raise ErasureError

    Return detached data followed by parity shards.

    Codec::max_shard_bytes

    source
    fn Codec::max_shard_bytes(self : Codec) -> Int

    Codec::new

    source
    fn Codec::new(data_count : Int, parity_count : Int, max_encoded_bytes? : Int) -> Codec raise ErasureError

    Codec::parity_count

    source
    fn Codec::parity_count(self : Codec) -> Int

    Codec::plan

    source
    fn Codec::plan(self : Codec, present : Array[Bool]) -> RepairPlan raise ErasureError

    Report the first k present shards chosen by deterministic index order. Storage adapters can fetch additional shards before allocating decode work.

    Codec::reconstruct

    source
    fn Codec::reconstruct(self : Codec, shards : Array[Bytes?]) -> Array[Bytes] raise ErasureError

    Reconstruct every erased shard from any data_count intact shards. Present shards are cross-checked against the reconstructed codeword. Exactly data_count present shards provide no spare evidence to detect an unknown corrupt byte; validate integrity envelopes before calling this API.

    Codec::total_count

    source
    fn Codec::total_count(self : Codec) -> Int

    Codec::verify

    source
    fn Codec::verify(self : Codec, shards : Array[Bytes]) -> Bool raise ErasureError

    Check whether every supplied parity shard agrees with the data shards.

    DecoderEntry

    source
    type DecoderEntry

    EncodedObject

    source
    pub struct EncodedObject {
      manifest : Manifest
      frames : Array[ShardEnvelope]
    }

    A bounded in-memory object ready for caller-controlled placement.

    EncodedObject::frame_count

    source
    fn EncodedObject::frame_count(self : EncodedObject) -> Int

    EncodedObject::frames

    source

    EncodedObject::manifest

    source
    fn EncodedObject::manifest(self : EncodedObject) -> Manifest

    Field

    source
    pub struct Field {
      exp : Array[Int]
      log : Array[Int]
    }

    GF(2^8) with irreducible polynomial 0x11d and generator 2. Keeping tables inside a codec avoids mutable global state across backends.

    FrameCatalog

    source
    pub struct FrameCatalog {
      manifest : Manifest
      slots : Array[ShardEnvelope?]
      stored_count : Int
    }

    In-memory index for independently stored frames of one manifest. A storage adapter may use it to decide which stripe to fetch or repair.

    FrameCatalog::add

    source
    fn FrameCatalog::add(self : FrameCatalog, frame : ShardEnvelope) -> Unit raise ErasureError

    Register one already parsed frame. Identity, dimensions, original stripe length and duplicate index are checked before the catalog changes.

    FrameCatalog::add_bytes

    source
    fn FrameCatalog::add_bytes(self : FrameCatalog, wire : Bytes, max_encoded_bytes? : Int) -> Unit raise ErasureError

    Parse a serialized frame and admit it only if both its CRC and catalog identity checks pass. A failed call leaves the catalog unchanged.

    FrameCatalog::add_many

    source
    fn FrameCatalog::add_many(self : FrameCatalog, frames : Array[ShardEnvelope], max_frames? : Int) -> Int raise ErasureError

    Admit a batch atomically. Validate on a detached slot table, then publish only after every identity and duplicate check succeeds. Frames are already parsed; use add_bytes for CRC validation of external serialized frames.

    FrameCatalog::capacity

    source
    fn FrameCatalog::capacity(self : FrameCatalog) -> Int

    FrameCatalog::get

    source
    fn FrameCatalog::get(self : FrameCatalog, stripe_index : Int, shard_index : Int) -> ShardEnvelope? raise ErasureError

    Look up one slot. A missing slot is returned as None.

    FrameCatalog::invalidate

    source
    fn FrameCatalog::invalidate(self : FrameCatalog, stripe_index : Int, shard_index : Int) -> Bool raise ErasureError

    Evict a slot after a storage read or checksum check shows that its bytes can no longer be trusted. Removing an absent slot is a harmless no-op.

    FrameCatalog::manifest

    source
    fn FrameCatalog::manifest(self : FrameCatalog) -> Manifest

    FrameCatalog::missing_indices

    source
    fn FrameCatalog::missing_indices(self : FrameCatalog, stripe_index : Int) -> Array[Int] raise ErasureError

    Return the missing shard indices of a stripe in ascending order.

    FrameCatalog::new

    source
    fn FrameCatalog::new(manifest : Manifest) -> FrameCatalog

    FrameCatalog::recover_range

    source
    fn FrameCatalog::recover_range(self : FrameCatalog, start : Int, length : Int, max_encoded_bytes? : Int) -> Bytes raise ErasureError

    Decode only the stripes intersecting a requested range from indexed frames. This is useful after an application has inventoried unordered stored fragments but needs a small portion of the original object.

    FrameCatalog::recover_stripe

    source
    fn FrameCatalog::recover_stripe(self : FrameCatalog, stripe_index : Int, max_encoded_bytes? : Int) -> StripeRecovery raise ErasureError

    Decode an indexed stripe. Present bytes are still cross-checked by recover_stripe, since catalog admission validates metadata only.

    FrameCatalog::recoverable_ranges

    source
    fn FrameCatalog::recoverable_ranges(self : FrameCatalog, max_ranges? : Int) -> Array[RecoverableRange] raise ErasureError

    Enumerate maximal contiguous ranges whose stripes each have at least k admitted frames. Supports partial media reads and resume scheduling without decoding the object. A result budget is checked before every emitted run.

    FrameCatalog::stored_count

    source
    fn FrameCatalog::stored_count(self : FrameCatalog) -> Int

    FrameCatalog::stripe_frames

    source
    fn FrameCatalog::stripe_frames(self : FrameCatalog, stripe_index : Int) -> Array[ShardEnvelope] raise ErasureError

    Return present frames of a stripe in canonical shard index order.

    FrameCatalog::stripe_recoverable

    source
    fn FrameCatalog::stripe_recoverable(self : FrameCatalog, stripe_index : Int) -> Bool raise ErasureError

    FrameIssue

    source
    pub struct FrameIssue {
      source_position : Int
      code : String
    }

    A rejected input frame is identified by its position in the input list. The shard index inside a damaged header is deliberately not trusted.

    FrameIssue::code

    source
    fn FrameIssue::code(self : FrameIssue) -> String

    FrameIssue::source_position

    source
    fn FrameIssue::source_position(self : FrameIssue) -> Int

    FrameScan

    source
    pub struct FrameScan {
      valid : Array[ShardEnvelope]
      issues : Array[FrameIssue]
    }

    FrameScan::issues

    source
    fn FrameScan::issues(self : FrameScan) -> Array[FrameIssue]

    FrameScan::rejected_count

    source
    fn FrameScan::rejected_count(self : FrameScan) -> Int

    FrameScan::valid

    source
    fn FrameScan::valid(self : FrameScan) -> Array[ShardEnvelope]

    FrameScan::valid_count

    source
    fn FrameScan::valid_count(self : FrameScan) -> Int

    Manifest

    source
    pub struct Manifest {
      set_id : Int
      data_count : Int
      parity_count : Int
      stripe_payload_limit : Int
      stripe_count : Int
      total_length : Int
    }

    Bounded description of an ordered sequence of independently coded stripes.

    Manifest::data_count

    source
    fn Manifest::data_count(self : Manifest) -> Int

    Manifest::from_bytes

    source
    fn Manifest::from_bytes(input : Bytes, max_object_bytes? : Int, max_encoded_bytes? : Int) -> Manifest raise ErasureError

    Manifest::new

    source
    fn Manifest::new(codec : Codec, set_id : Int, stripe_payload_limit : Int, total_length : Int, max_object_bytes? : Int) -> Manifest raise ErasureError

    Manifest::parity_count

    source
    fn Manifest::parity_count(self : Manifest) -> Int

    Manifest::set_id

    source
    fn Manifest::set_id(self : Manifest) -> Int

    Manifest::stripe_count

    source
    fn Manifest::stripe_count(self : Manifest) -> Int

    Manifest::stripe_length

    source
    fn Manifest::stripe_length(self : Manifest, index : Int) -> Int raise ErasureError

    Manifest::stripe_payload_limit

    source
    fn Manifest::stripe_payload_limit(self : Manifest) -> Int

    Manifest::to_bytes

    source
    fn Manifest::to_bytes(self : Manifest) -> Bytes

    28 bytes of metadata followed by CRC32C of those 28 bytes.

    Manifest::total_length

    source
    fn Manifest::total_length(self : Manifest) -> Int

    Matrix

    source
    pub struct Matrix {
      rows : Int
      cols : Int
      cells : Array[Array[Int]]
    }

    Small field matrix used only for codec construction and erasure recovery.

    ObjectDecoder

    source
    pub struct ObjectDecoder {
      codec : Codec
      manifest : Manifest
      next_stripe : Int
      decoded_bytes : Int
      repaired_shards : Int
    }

    Sequential decoder for objects whose frames are fetched stripe by stripe. The caller groups each stripe's available frames and can write each returned payload chunk directly to a destination.

    ObjectDecoder::decoded_bytes

    source
    fn ObjectDecoder::decoded_bytes(self : ObjectDecoder) -> Int

    ObjectDecoder::finish

    source
    fn ObjectDecoder::finish(self : ObjectDecoder) -> Int raise ErasureError

    Confirm that every stripe was consumed and the advertised byte count matches. There is no hidden payload buffer in this decoder.

    ObjectDecoder::new

    source
    fn ObjectDecoder::new(manifest : Manifest, max_encoded_bytes? : Int) -> ObjectDecoder raise ErasureError

    ObjectDecoder::next_stripe

    source
    fn ObjectDecoder::next_stripe(self : ObjectDecoder) -> Int

    ObjectDecoder::push_stripe

    source
    fn ObjectDecoder::push_stripe(self : ObjectDecoder, frames : Array[ShardEnvelope]) -> Bytes raise ErasureError

    Decode exactly the next manifest stripe. On failure, the cursor remains unchanged so callers may fetch an alternative frame and retry.

    ObjectDecoder::repaired_shards

    source
    fn ObjectDecoder::repaired_shards(self : ObjectDecoder) -> Int

    ObjectEncoder

    source
    pub struct ObjectEncoder {
      codec : Codec
      manifest : Manifest
      pending : Array[Byte]
      accepted_bytes : Int
      emitted_stripes : Int
    }

    Incremental object encoder with an upfront manifest. Applications can persist completed frames immediately, keeping only one stripe in memory.

    ObjectEncoder::accepted_bytes

    source
    fn ObjectEncoder::accepted_bytes(self : ObjectEncoder) -> Int

    ObjectEncoder::buffered_bytes

    source
    fn ObjectEncoder::buffered_bytes(self : ObjectEncoder) -> Int

    ObjectEncoder::emitted_stripes

    source
    fn ObjectEncoder::emitted_stripes(self : ObjectEncoder) -> Int

    ObjectEncoder::finish

    source
    fn ObjectEncoder::finish(self : ObjectEncoder) -> Manifest raise ErasureError

    Confirm the stream ended exactly at the declared length and every stripe was emitted. Empty objects finish without emitting frames.

    ObjectEncoder::manifest

    source
    fn ObjectEncoder::manifest(self : ObjectEncoder) -> Manifest

    ObjectEncoder::new

    source
    fn ObjectEncoder::new(codec : Codec, set_id : Int, total_length : Int, stripe_payload_limit? : Int, max_object_bytes? : Int) -> ObjectEncoder raise ErasureError

    ObjectEncoder::push

    source
    fn ObjectEncoder::push(self : ObjectEncoder, chunk : Bytes) -> Array[ShardEnvelope] raise ErasureError

    Accept a chunk of the declared object. A chunk may complete zero, one, or many stripes. Exceeding the advertised size is rejected before mutation.

    ObjectHealth

    source
    pub struct ObjectHealth {
      stripes : Array[StripeHealth]
      healthy_count : Int
      repairable_count : Int
      unrecoverable_count : Int
      invalid_count : Int
    }

    ObjectHealth::fully_recoverable

    source
    fn ObjectHealth::fully_recoverable(self : ObjectHealth) -> Bool

    ObjectHealth::healthy_count

    source
    fn ObjectHealth::healthy_count(self : ObjectHealth) -> Int

    ObjectHealth::invalid_count

    source
    fn ObjectHealth::invalid_count(self : ObjectHealth) -> Int

    ObjectHealth::repairable_count

    source
    fn ObjectHealth::repairable_count(self : ObjectHealth) -> Int

    ObjectHealth::stripes

    source

    ObjectHealth::unrecoverable_count

    source
    fn ObjectHealth::unrecoverable_count(self : ObjectHealth) -> Int

    ObjectRecovery

    source
    pub struct ObjectRecovery {
      payload : Bytes
      repaired_shard_count : Int
      stripe_count : Int
    }

    ObjectRecovery::payload

    source
    fn ObjectRecovery::payload(self : ObjectRecovery) -> Bytes

    ObjectRecovery::repaired_shard_count

    source
    fn ObjectRecovery::repaired_shard_count(self : ObjectRecovery) -> Int

    ObjectRecovery::stripe_count

    source
    fn ObjectRecovery::stripe_count(self : ObjectRecovery) -> Int

    ObjectRepair

    source
    pub struct ObjectRepair {
      replacements : Array[ShardEnvelope]
      repaired_stripes : Int
    }

    Replacement frames for missing slots, in stripe then shard order. The caller decides where to persist them. Existing frames are never rewritten.

    ObjectRepair::repaired_stripes

    source
    fn ObjectRepair::repaired_stripes(self : ObjectRepair) -> Int

    ObjectRepair::replacement_count

    source
    fn ObjectRepair::replacement_count(self : ObjectRepair) -> Int

    ObjectRepair::replacements

    source
    fn ObjectRepair::replacements(self : ObjectRepair) -> Array[ShardEnvelope]

    RecoverableRange

    source
    pub struct RecoverableRange {
      start : Int
      end : Int
    }

    A half-open byte range that has enough admitted frames to attempt recovery. Presence is not authentication or a new content checksum verification.

    RecoverableRange::end

    source
    fn RecoverableRange::end(self : RecoverableRange) -> Int

    RecoverableRange::length

    source
    fn RecoverableRange::length(self : RecoverableRange) -> Int

    RecoverableRange::start

    source
    fn RecoverableRange::start(self : RecoverableRange) -> Int

    RecoverySession

    source
    pub struct RecoverySession {
      codec : Codec
      entries : Array[DecoderEntry]
      capacity : Int
      next_evict : Int
      hits : Int
      misses : Int
    }

    Reuse decode matrices when many stripes have the same erasure pattern. Each session owns a small bounded FIFO cache; separate sessions should be used for concurrent callers. Cached matrices contain no payload bytes.

    RecoverySession::cache_hits

    source
    fn RecoverySession::cache_hits(self : RecoverySession) -> Int

    RecoverySession::cache_misses

    source
    fn RecoverySession::cache_misses(self : RecoverySession) -> Int

    RecoverySession::cache_size

    source
    fn RecoverySession::cache_size(self : RecoverySession) -> Int

    RecoverySession::new

    source
    fn RecoverySession::new(codec : Codec, max_patterns? : Int) -> RecoverySession raise ErasureError

    RecoverySession::reconstruct

    source
    fn RecoverySession::reconstruct(self : RecoverySession, shards : Array[Bytes?]) -> Array[Bytes] raise ErasureError

    Reconstruct with the same validation as Codec::reconstruct. Data-only stripes skip matrix work and do not alter cache counters. Cache misses calculate a dense inverse once and reuse it on subsequent matching stripes.

    RepairPlan

    source
    pub struct RepairPlan {
      required : Int
      available : Int
      selected_indices : Array[Int]
      missing_data_indices : Array[Int]
      missing_parity_indices : Array[Int]
    }

    A deterministic preflight for a pattern of present and erased shards. It does not read shard bytes or make an integrity assertion.

    RepairPlan::available

    source
    fn RepairPlan::available(self : RepairPlan) -> Int

    RepairPlan::minimum_additional_shards

    source
    fn RepairPlan::minimum_additional_shards(self : RepairPlan) -> Int

    RepairPlan::missing_data_indices

    source
    fn RepairPlan::missing_data_indices(self : RepairPlan) -> Array[Int]

    RepairPlan::missing_parity_indices

    source
    fn RepairPlan::missing_parity_indices(self : RepairPlan) -> Array[Int]

    RepairPlan::needs_decode

    source
    fn RepairPlan::needs_decode(self : RepairPlan) -> Bool

    RepairPlan::recoverable

    source
    fn RepairPlan::recoverable(self : RepairPlan) -> Bool

    RepairPlan::required

    source
    fn RepairPlan::required(self : RepairPlan) -> Int

    RepairPlan::selected_indices

    source
    fn RepairPlan::selected_indices(self : RepairPlan) -> Array[Int]

    ScannedObjectHealth

    source
    pub struct ScannedObjectHealth {
      health : ObjectHealth
      issues : Array[FrameIssue]
    }

    Health inspection of raw frames: invalid encodings are reported by input position, while stripe status is computed from surviving valid frames.

    ScannedObjectHealth::health

    source

    ScannedObjectHealth::issues

    source

    ScannedObjectRecovery

    source
    pub struct ScannedObjectRecovery {
      recovery : ObjectRecovery
      issues : Array[FrameIssue]
    }

    Object recovery together with diagnostics for rejected raw frame positions.

    ScannedObjectRecovery::issues

    source

    ScannedObjectRecovery::recovery

    source

    ScannedObjectRepair

    source
    pub struct ScannedObjectRepair {
      repair : ObjectRepair
      issues : Array[FrameIssue]
    }

    Repair result for raw frame inputs. Damaged frames are reported by input position and treated as absent; replacement frames are checksum-protected.

    ScannedObjectRepair::issues

    source

    ScannedObjectRepair::repair

    source

    ScannedRangeRecovery

    source
    pub struct ScannedRangeRecovery {
      payload : Bytes
      issues : Array[FrameIssue]
    }

    A recovered byte range and diagnostics for raw frames rejected by CRC or structural validation.

    ScannedRangeRecovery::issues

    source

    ScannedRangeRecovery::payload

    source
    fn ScannedRangeRecovery::payload(self : ScannedRangeRecovery) -> Bytes

    ScannedRecovery

    source
    pub struct ScannedRecovery {
      recovery : StripeRecovery
      issues : Array[FrameIssue]
    }

    ScannedRecovery::issues

    source

    ScannedRecovery::recovery

    source

    ShardEnvelope

    source
    pub struct ShardEnvelope {
      set_id : Int
      stripe_index : Int
      shard_index : Int
      data_count : Int
      parity_count : Int
      original_length : Int
      payload : Bytes
    }

    Version 1 frame: 28-byte metadata, 4-byte CRC32C, then shard payload. Integer fields are unsigned little-endian; ids and lengths are capped at signed 31-bit values so they behave identically on all MoonBit targets.

    ShardEnvelope::data_count

    source
    fn ShardEnvelope::data_count(self : ShardEnvelope) -> Int

    ShardEnvelope::from_bytes

    source
    fn ShardEnvelope::from_bytes(input : Bytes, max_encoded_bytes? : Int) -> ShardEnvelope raise ErasureError

    Parse one bounded frame. Invalid metadata and damaged payloads fail closed.

    ShardEnvelope::new

    source
    fn ShardEnvelope::new(codec : Codec, set_id : Int, stripe_index : Int, shard_index : Int, original_length : Int, payload : Bytes) -> ShardEnvelope raise ErasureError

    ShardEnvelope::original_length

    source
    fn ShardEnvelope::original_length(self : ShardEnvelope) -> Int

    ShardEnvelope::parity_count

    source
    fn ShardEnvelope::parity_count(self : ShardEnvelope) -> Int

    ShardEnvelope::payload

    source
    fn ShardEnvelope::payload(self : ShardEnvelope) -> Bytes

    ShardEnvelope::set_id

    source
    fn ShardEnvelope::set_id(self : ShardEnvelope) -> Int

    ShardEnvelope::shard_index

    source
    fn ShardEnvelope::shard_index(self : ShardEnvelope) -> Int

    ShardEnvelope::stripe_index

    source
    fn ShardEnvelope::stripe_index(self : ShardEnvelope) -> Int

    ShardEnvelope::to_bytes

    source
    fn ShardEnvelope::to_bytes(self : ShardEnvelope) -> Bytes

    Serialize a validated envelope with a CRC covering metadata and payload.

    StripeHealth

    source
    pub struct StripeHealth {
      stripe_index : Int
      status : StripeStatus
      present_count : Int
      missing_indices : Array[Int]
      reason_code : String?
    }

    StripeHealth::missing_indices

    source
    fn StripeHealth::missing_indices(self : StripeHealth) -> Array[Int]

    StripeHealth::present_count

    source
    fn StripeHealth::present_count(self : StripeHealth) -> Int

    StripeHealth::reason_code

    source
    fn StripeHealth::reason_code(self : StripeHealth) -> String?

    StripeHealth::status

    source
    fn StripeHealth::status(self : StripeHealth) -> StripeStatus

    StripeHealth::stripe_index

    source
    fn StripeHealth::stripe_index(self : StripeHealth) -> Int

    StripeRecovery

    source
    pub struct StripeRecovery {
      payload : Bytes
      shards : Array[ShardEnvelope]
      missing_indices : Array[Int]
      provided_count : Int
    }

    A complete repaired stripe plus information about which shards were absent.

    StripeRecovery::missing_indices

    source
    fn StripeRecovery::missing_indices(self : StripeRecovery) -> Array[Int]

    StripeRecovery::payload

    source
    fn StripeRecovery::payload(self : StripeRecovery) -> Bytes

    StripeRecovery::provided_count

    source
    fn StripeRecovery::provided_count(self : StripeRecovery) -> Int

    StripeRecovery::shards

    source

    StripeStatus

    source
    pub(all) enum StripeStatus {
      Healthy
      Repairable
      Unrecoverable
      InvalidInput
    }

    Result of examining one stripe without returning payload bytes.

    StripeStatus::label

    source
    fn StripeStatus::label(self : StripeStatus) -> String

    crc32c

    source
    fn crc32c(input : Bytes) -> UInt

    encode_object

    source
    fn encode_object(codec : Codec, payload : Bytes, set_id : Int, stripe_payload_limit? : Int, max_object_bytes? : Int, max_output_bytes? : Int) -> EncodedObject raise ErasureError

    Split an object into bounded, independently recoverable stripes.

    encode_stripe

    source
    fn encode_stripe(codec : Codec, payload : Bytes, set_id : Int, stripe_index : Int) -> Array[ShardEnvelope] raise ErasureError

    Split one nonempty payload into equal padded data shards and encode parity.

    export_bundle

    source
    fn export_bundle(object : EncodedObject, max_output_bytes? : Int) -> Bytes raise ErasureError

    Single-file archive: 24-byte bundle header, 32-byte manifest, then a length-prefixed sequence of complete shard frames. Individual frames retain their own checksums so they can also be extracted and stored independently.

    import_bundle

    source
    fn import_bundle(input : Bytes, max_input_bytes? : Int, max_object_bytes? : Int, max_encoded_bytes? : Int) -> EncodedObject raise ErasureError

    Strictly import one complete bundle and validate every stripe. This archive path refuses damage; individual frame recovery remains available through recover_serialized_stripe when some bytes are unavailable.

    inspect_object

    source
    fn inspect_object(manifest : Manifest, frames : Array[ShardEnvelope], max_encoded_bytes? : Int) -> ObjectHealth raise ErasureError

    Inspect every stripe against a manifest. The caller may decide which repairable stripes to reconstruct and where to place replacement shards.

    inspect_serialized_object

    source
    fn inspect_serialized_object(manifest : Manifest, inputs : Array[Bytes], max_encoded_bytes? : Int, max_total_bytes? : Int) -> ScannedObjectHealth raise ErasureError

    Inspect an object directly from independently stored frame bytes. A bad checksum is counted as missing capacity, but a valid frame with conflicting set id, dimensions or duplicate slot marks input invalid.

    inspect_stripe

    source
    fn inspect_stripe(codec : Codec, frames : Array[ShardEnvelope], set_id : Int, stripe_index : Int) -> StripeHealth

    Plan an individual stripe and verify consistency when enough frames exist.

    recover_object

    source
    fn recover_object(manifest : Manifest, frames : Array[ShardEnvelope], max_encoded_bytes? : Int) -> ObjectRecovery raise ErasureError

    Reassemble in manifest order; every stripe must have enough intact frames. For individual repaired shard bytes, call recover_stripe on that stripe.

    recover_object_range

    source
    fn recover_object_range(manifest : Manifest, frames : Array[ShardEnvelope], start : Int, length : Int, max_encoded_bytes? : Int) -> Bytes raise ErasureError

    Recover only the stripes overlapping a byte range. Callers may supply frames for the whole object or only for the requested stripes. Damage in unrelated stripes does not block a range read.

    recover_serialized_object

    source
    fn recover_serialized_object(manifest : Manifest, inputs : Array[Bytes], max_encoded_bytes? : Int, max_total_bytes? : Int) -> ScannedObjectRecovery raise ErasureError

    Recover an object from independently received shard frames. Corrupt or malformed frames become known erasures; a mismatch among otherwise valid frames is still rejected by stripe reconstruction.

    recover_serialized_range

    source
    fn recover_serialized_range(manifest : Manifest, inputs : Array[Bytes], start : Int, length : Int, max_encoded_bytes? : Int, max_total_bytes? : Int) -> ScannedRangeRecovery raise ErasureError

    Recover an object byte range from independently stored serialized frames. Only intersecting stripes are decoded. Damaged frames within those stripes are erasures and must fit the parity budget.

    recover_serialized_stripe

    source
    fn recover_serialized_stripe(codec : Codec, inputs : Array[Bytes], set_id : Int, stripe_index : Int) -> ScannedRecovery raise ErasureError

    Treat checksum-failed or malformed frames as erasures, while retaining a diagnostic for each rejected input position. A sufficient set of valid frames is still required.

    recover_stripe

    source
    fn recover_stripe(codec : Codec, frames : Array[ShardEnvelope], set_id : Int, stripe_index : Int) -> StripeRecovery raise ErasureError

    Recover a stripe from validated envelopes. Duplicate indices, mixed identities, mismatched sizes and inconsistent extra shards are rejected.

    repair_object

    source
    fn repair_object(manifest : Manifest, frames : Array[ShardEnvelope], max_encoded_bytes? : Int) -> ObjectRepair raise ErasureError

    Regenerate only absent frames. Every supplied frame is checked by the stripe decoder; missing frames beyond parity capacity abort the operation. This operation is all-or-error: callers receive no partial repair list.

    repair_serialized_object

    source
    fn repair_serialized_object(manifest : Manifest, inputs : Array[Bytes], max_encoded_bytes? : Int, max_total_bytes? : Int) -> ScannedObjectRepair raise ErasureError

    Scan independent serialized frames and regenerate every absent or damaged slot. The returned issue positions let storage adapters quarantine bad objects without trusting their internal shard indices.

    scan_frames

    source
    fn scan_frames(inputs : Array[Bytes], max_frames? : Int, max_encoded_bytes? : Int, max_total_bytes? : Int) -> FrameScan raise ErasureError

    Parse independent frames without discarding good neighbors. The caller must still validate stripe identity and duplicates when using the valid frames.

    Source Files

    Powered by MoonBit

    Site sourceReport issuePackagesBuild queueSkillsStatistics

    © 2026 mooncakes.io