🥮 mooncakes.io

    #core/persistence/wal

    Write-Ahead Log (WAL) for crash recovery.

    #Format

    Header (12 bytes):
  - Magic: "VCDBMBT\0" (8 bytes)
  - Type: 1 byte
  - Version: 1 byte
  - Reserved: 2 bytes

Records:
  - Type: 1 byte (Upsert/Remove/SetAttrs)
  - Reserved: 1 byte
  - ID: 8 bytes
  - AttrsLen: 4 bytes
  - VectorLen: 4 bytes
  - Attrs: JSON UTF-8
  - Vector: Float32 array

Footer (8 bytes, optional):
  - Magic: "WCRC" (4 bytes)
  - CRC32: 4 bytes

    #Record Types

    • Upsert: Insert or update vector with attrs
    • Remove: Delete vector
    • SetAttrs: Update attrs only

    #Usage

    let writer = WalWriter::new()
    writer.write_upsert(id, vector, attrs)
    let bytes = writer.finalize()

    AsyncWalRuntime

    source
    pub struct AsyncWalRuntime[S] {
      storage : S
      path : String
      kind : 
    StorageKind
      data : Bytes
      record_count : Int
      loaded : Bool
    }

    AsyncWalRuntime::append

    source

    Append records to the WAL. Merges in-memory (no read from storage), then writes to storage. Delegates merge logic to format.mbt's merge_wal.

    AsyncWalRuntime::async_exists

    source

    Check if WAL exists on storage.

    AsyncWalRuntime::byte_size

    source
    fn[S] AsyncWalRuntime::byte_size(self : AsyncWalRuntime[S]) -> Int

    Current WAL data size in bytes (for byte-size checkpoint decisions).

    AsyncWalRuntime::is_loaded

    source
    fn[S] AsyncWalRuntime::is_loaded(self : AsyncWalRuntime[S]) -> Bool

    Whether load() has been called.

    AsyncWalRuntime::load

    source

    Load WAL data from storage into the in-memory buffer. Must be called once before append/truncate. Idempotent. Returns the WAL data bytes (for callers that need it, e.g., replay).

    AsyncWalRuntime::new

    source
    fn[S] AsyncWalRuntime::new(storage : S, path : String, kind? : 
    StorageKind    ) -> AsyncWalRuntime[S]

    AsyncWalRuntime::path

    source
    fn[S] AsyncWalRuntime::path(self : AsyncWalRuntime[S]) -> String

    Get the WAL file path.

    AsyncWalRuntime::record_count

    source
    fn[S] AsyncWalRuntime::record_count(self : AsyncWalRuntime[S]) -> Int

    Number of records in the current WAL buffer (since last truncate).

    AsyncWalRuntime::replay_into

    source

    Replay WAL records into a CoreStore. Returns number of records applied. Delegates replay logic to replay_wal_data in runtime.mbt.

    Uses in-memory buffer if loaded, otherwise reads from storage.

    AsyncWalRuntime::truncate

    source

    Truncate the WAL — reset in-memory buffer and write empty header.

    WalRecord

    source
    pub struct WalRecord {
      record_type : WalRecordType
      id : 
    VectorId
      attrs : 
    Attrs    ?
      vector : Array[Double]?
      timestamp : Int64
    } derive(
    Debug    )

    WAL record with timestamp
    impl Show for WalRecord

    WalRecord::remove

    source
    fn WalRecord::remove(id : 
    VectorId    , timestamp? : Int64) -> WalRecord

    Create a remove record with timestamp

    WalRecord::set_attrs

    source

    Create a set_attrs record with timestamp

    WalRecord::upsert

    source
    fn WalRecord::upsert(id : 
    VectorId    , vector : Array[Double], attrs : 
    Attrs    , timestamp? : Int64) -> WalRecord

    Create an upsert record with timestamp

    WalRecordType

    source
    pub enum WalRecordType {
      Upsert
      Remove
      SetAttrs
    } derive(Eq, 
    Debug    )

    WAL record types

    WalRecordType::from_byte

    source
    fn WalRecordType::from_byte(b : Byte) -> WalRecordType?

    WalRecordType::to_byte

    source
    fn WalRecordType::to_byte(self : WalRecordType) -> Byte

    decode_wal_record

    source

    Decode a WAL record from reader (public entry point — assumes v3 format). For reading v1/v2 WAL files, use decode_wal_records which handles versioning.

    decode_wal_records

    source
    fn decode_wal_records(data : Bytes) -> Array[WalRecord]

    Decode all records from WAL data (supports v1, v2, and v3)
    fn encode_wal_footer(body : Bytes) -> Bytes

    Encode WAL footer with CRC32

    encode_wal_header

    source
    fn encode_wal_header() -> Bytes

    Encode WAL header (12 bytes)

    encode_wal_record

    source
    fn encode_wal_record(record : WalRecord) -> Bytes

    Encode a WAL record to bytes (v2 format with timestamp)

    encode_wal_segment

    source
    fn encode_wal_segment(records : Array[WalRecord]) -> Bytes

    Encode multiple records to a complete WAL segment

    filter_wal_by_timestamp

    source
    fn filter_wal_by_timestamp(data : Bytes, cutoff_ts : Int64) -> Array[WalRecord]

    Filter WAL records by timestamp — returns only records with timestamp <= cutoff. Used for read-at-timestamp (consistent cross-shard reads).

    max_wal_timestamp

    source
    fn max_wal_timestamp(data : Bytes) -> Int64

    Get the maximum timestamp across all records in a WAL. Returns 0 if no timestamped records.

    merge_wal

    source
    fn merge_wal(existing : Bytes, new_segment : Bytes) -> Bytes

    Merge an existing WAL with a new WAL segment.

    replay_wal_data

    source
    fn replay_wal_data(data : Bytes, store : 
    CoreStore    , cutoff_ts? : Int64) -> Int

    Replay WAL binary data into a CoreStore (no Storage dependency). Used by:
    • AsyncWalRuntime::replay_into (reads from AsyncStorage, then delegates here)
    • JS persistent exports (passes WAL bytes directly)

    Returns the number of records applied. Returns 0 for empty/invalid data.

    verify_wal_checksum

    source
    fn verify_wal_checksum(data : Bytes) -> Bool

    Check if WAL has valid footer and verify CRC32

    verify_wal_header

    source
    fn verify_wal_header(r : 
    BinaryReader    ) -> Bool

    Verify WAL header (accepts v1, v2, and v3). Returns the version byte on success.
    let wal_footer_magic : UInt

    WAL Footer magic: "WCRC" in little-endian

    wal_records_for_replay

    source
    fn wal_records_for_replay(data : Bytes, cutoff_ts? : Int64) -> Array[WalRecord]

    Decode replayable WAL records after validating header and checksum.

    wal_version

    source
    let wal_version : Byte

    Current WAL format version

    Source Files

    Powered by MoonBit

    Site sourceReport issueModulesSkillsCLIBuild queueStatistics

    © 2026 mooncakes.io