🥮 mooncakes.io

    desert

    Binary serialization with schema evolution, wire-compatible with desert (Scala) and desert-rust

    serialization
    binary
    codec
    evolution
    moonbit
    Download zip
    Author
    Version
    0.1.0
    License
    Apache-2.0
    Repository
    Last updated
    5 hours ago
    Downloads
    3

    No dependencies

    #desert for MoonBit

    A MoonBit implementation of the desert binary serialization format, wire-compatible with the original Scala library and desert-rust.

    Documentation: see the docs/ chapters — getting started, codecs, schema evolution, wire format reference, input/output, type registry, benchmarks.

    See my a conference talk from 2022 about the original Scala version.

    #Features

    • Byte-compatible wire format with desert (Scala) and desert-rust, pinned by the shared golden dataset (dataset1.bin, 242,540 bytes written by the Scala library) and byte-exact cross-implementation dumps
    • Schema evolution: versioned records with field add/remove/ make-optional/make-transient, forward and backward compatible reads, and unknown-step tolerance
    • Code generation: #desert(binary_codec) annotations produce codec implementations via the MoonBit-native desert-codegen tool (parse → validate → format pipeline, idempotent output)
    • Opt-in string deduplication (DeduplicatedString) with session-scoped ids, shared with the evolution headers
    • Type registry for runtime-dispatched ("unknown") serialization
    • Known limitation: object back-reference tracking (Scala storeRefOrObject) is not implemented yet; see docs/codecs.md
    • Compressed byte blocks (DEFLATE via moonbitlang/core/compress)
    • Pure MoonBit, no FFI — runs on wasm, wasm-gc, js, and native
    • Property-based tests (moonbitlang/quickcheck) and micro benchmarks (moon bench) — see the desert-rust comparison report

    #Usage

    // Serialize / deserialize (raises @desert.DesertFailure on errors)

    ///|
    let bytes = @desert.serialize_to_bytes(value)

    ///|
    let back : MyType = @desert.deserialize_exhaustive(bytes)

    // Derive codecs for your types

    ///|
    #desert(binary_codec, evolution(FieldAdded("extra", "default")))
    struct MyRecord {
      base : Int32
      extra : String
    }

    # from a clone of this repository (a MoonBit workspace):
moon run codegen -- <your-package-directory>

    #Project layout

    MoonBit organizes code into packages — each directory with a moon.pkg is one package and one namespace, imported as @desert (or vigoo/desert/codegen for subpackages). Files within a package are pure organization: they do not define namespaces (unlike Java/C#/Rust modules), and splitting code across topical files is the standard idiom — see moonbitlang/core/builtin (241 .mbt files) and moonbitlang/quickcheck in the standard ecosystem. This repository follows that shape:

    moon.mod            module metadata (published as vigoo/desert)
moon.pkg            the root package — the library itself
error.mbt           DesertFailure taxonomy
binary_input.mbt    read cursor (BinaryInput)
binary_output.mbt   write sink (BinaryOutput + ByteOutput)
codecs_*.mbt        primitive/string/collection/tuple codecs
evolution.mbt       evolution metadata + wire codes
adt_writer.mbt      evolved-record writer engine
adt_reader.mbt      evolved-record reader engine
state.mbt           per-session dedup state
registry.mbt        type registry
compressed.mbt      DEFLATE byte blocks
facade.mbt          top-level serialize/deserialize
*_test.mbt / *_wbtest.mbt   blackbox / whitebox tests
codegen/            desert-codegen generator (executable package)
codegen_demo/       generated-code golden test package
docs/               documentation chapters

    The generated pkg.generated.mbti file is the package's public API surface (the equivalent of Rust's lib.rs exports — run moon info to refresh it).

    #Development

    moon check                  # type check (0 warnings policy)
moon test                   # byte-exact, evolution, registry, property tests
moon bench                  # micro benchmarks
moon run codegen -- codegen_demo   # regenerate the demo codecs
moon info && moon fmt       # refresh interface + format

    BinaryCodec

    source
    pub(open) trait BinaryCodec {
    }

    A type that can be both serialized and deserialized. Implement this (together with the two component traits) for your own types, or let desert-codegen generate the implementations.

    BinaryDeserializer

    source
    pub(open) trait BinaryDeserializer {
      fn deserialize(ctx : DeserializationContext) -> Self raise DesertFailure
    }

    Deserialization mirrors BinarySerializer: the method returns the decoded value of its own type.
    impl BinaryDeserializer for Map[K, V]
    impl BinaryDeserializer for Result[T, E]
    impl BinaryDeserializer for Tuple2[A, B]
    impl BinaryDeserializer for Tuple3[A, B, C]
    impl BinaryDeserializer for Tuple4[A, B, C, D]
    impl BinaryDeserializer for Tuple5[A, B, C, D, E]
    impl BinaryDeserializer for Tuple6[A, B, C, D, E, F]

    BinaryOutput

    source
    pub(open) trait BinaryOutput {
      fn write_u8(Self, Byte) -> Unit raise DesertFailure
      fn write_bytes(Self, BytesView) -> Unit raise DesertFailure
      fn write_i8(Self, Int) -> Unit raise DesertFailure = _
      fn write_u16(Self, UInt16) -> Unit raise DesertFailure = _
      fn write_i16(Self, Int) -> Unit raise DesertFailure = _
      fn write_u32(Self, UInt) -> Unit raise DesertFailure = _
      fn write_i32(Self, Int) -> Unit raise DesertFailure = _
      fn write_u64(Self, UInt64) -> Unit raise DesertFailure = _
      fn write_i64(Self, Int64) -> Unit raise DesertFailure = _
      fn write_f32(Self, Float) -> Unit raise DesertFailure = _
      fn write_f64(Self, Double) -> Unit raise DesertFailure = _
      fn write_var_u32(Self, UInt) -> Unit raise DesertFailure = _
      fn write_var_i32(Self, Int) -> Unit raise DesertFailure = _
      fn write_string_utf8(Self, StringView) -> Unit raise DesertFailure = _
    }

    A byte sink the serializer writes to.

    Only the two raw operations are required; everything else (fixed-width big-endian primitives, varints, floats) has a default implementation built on them, so any custom sink only has to provide write_u8 and write_bytes. The built-in ByteOutput overrides the fixed-width methods with bulk fast paths.

    The wire format is fixed: all fixed-width values are big-endian, varints are LEB128-style base-128 (least-significant 7 bits first, high bit is the continuation bit) with zigzag mapping for the signed variant.

    BinarySerializer

    source
    pub(open) trait BinarySerializer {
      fn serialize(Self, ctx : SerializationContext) -> Unit raise DesertFailure
    }

    Serialization is implemented as a single method taking the context; failures are raised as DesertFailure.
    impl BinarySerializer for Map[K, V]
    impl BinarySerializer for Result[T, E]
    impl BinarySerializer for HashMap[K, V]
    impl BinarySerializer for Tuple2[A, B]
    impl BinarySerializer for Tuple3[A, B, C]
    impl BinarySerializer for Tuple4[A, B, C, D]
    impl BinarySerializer for Tuple5[A, B, C, D, E]
    impl BinarySerializer for Tuple6[A, B, C, D, E, F]

    RegisteredValue

    source
    pub(open) trait RegisteredValue {
      fn registered_type_name(Self) -> String
      fn serialize_registered(Self, SerializationContext) -> Unit raise DesertFailure
    }

    Trait implemented by every type that can travel through the type registry as an erased value.

    Implementations are typically generated by desert-codegen (type_id = "pkg.TypeName"); hand-written ones just need a unique, stable type name and a delegate to the type's codec.

    DesertFailure

    source
    pub(all) suberror DesertFailure {
      FailedToReadInput(String)
      InputEndedUnexpectedly
      FailedToWriteOutput(String)
      SerializationFailure(String)
      DeserializationFailure(String)
      DeserializingNonExistingChunk(Int)
      NonOptionalFieldSerializedAsNone(String)
      UnknownFieldReferenceInEvolutionStep(String)
      UnknownSerializedEvolutionStep(Int)
      InvalidStringId(Int)
      FieldRemovedInSerializedVersion(String)
      FieldWithoutDefaultValueIsMissing(String)
      TypeNotRegistered(String)
      InvalidTypeId(Int)
      SerializationUpcastError(String, String, String)
      InvalidConstructorName(String, String)
      InvalidConstructorId(Int, String)
      InvalidRefId(Int)
      SerializingTransientConstructor(String)
      UnsupportedCharacter(Char)
      FailedToDecodeCharacter(UInt16)
    } derive(
    Debug    )

    The failure taxonomy of desert.

    Byte-for-byte mirror of the DesertFailure sealed trait of the original Scala library (and the Error enum of desert-rust): every variant that can appear on a wire-relevant code path exists here, so failure identities round-trip across implementations.

    Causes are flattened into the message payload; the original library carried an optional cause exception which has no MoonBit counterpart.

    DesertFailure::message

    source
    fn DesertFailure::message(self : DesertFailure) -> String

    Human-readable message, mirroring DesertFailure.message of the Scala library (wording kept close to the original where it exists).

    DesertFailure::output

    source
    fn DesertFailure::output(self : DesertFailure, logger : &Logger) -> Unit

    DesertFailure::to_repr

    source

    DesertFailure::to_string

    source
    fn DesertFailure::to_string(self : DesertFailure) -> String

    AdtMetadata

    source
    pub struct AdtMetadata {
      // private fields
    }

    Static description of a record's evolution history, shared by the writer and the reader. Build once per record type (e.g. as a toplevel let) via AdtMetadata::new.

    AdtMetadata::new

    source
    fn AdtMetadata::new(steps : Array[Evolution]) -> AdtMetadata

    Build the metadata for a record declared with steps (InitialVersion first, then one entry per schema change).

    Panics with more than 255 steps (the version byte is a single u8).

    AdtReader

    source
    pub struct AdtReader {
      // private fields
    }

    Reader for evolved records, the runtime engine behind derived codecs (mirrors desert-rust's AdtDeserializer).

    The generated (or hand-written) codec first reads the stored version byte, then either uses the direct field reads (both versions are 0) or constructs this reader, which parses the evolution header, fences each chunk behind an input region, and applies the compatibility rules:

    • fields introduced after the stored version fall back to their FieldAdded defaults (or fail without one)
    • fields removed in the serialized version fail on read (FieldRemovedInSerializedVersion) — optional fields return None
    • a field made optional after the stored version is read bare and wrapped in Some
    • an old reader finding a made-optional field serialized as None fails with NonOptionalFieldSerializedAsNone
    • unknown header entries (code 0) from future versions are ignored

    AdtReader::new

    source
    fn AdtReader::new(metadata : AdtMetadata, ctx : DeserializationContext, stored_version : Int) -> AdtReader raise DesertFailure

    Create a reader for data stored with stored_version. Parses the stored_version + 1 header entries and slices the input into per-version chunk regions.

    AdtReader::new_v0

    source
    fn AdtReader::new_v0(metadata : AdtMetadata, ctx : DeserializationContext) -> AdtReader

    Create a reader for data whose stored version is 0: fields follow the version byte directly (the metadata may still describe a newer schema — new readers reading old data).

    AdtReader::read_constructor_idx

    source
    fn AdtReader::read_constructor_idx(self : AdtReader) -> Int raise DesertFailure

    Read the coproduct constructor index (the record itself being an enum).

    AdtReader::read_field

    source
    fn[T : BinaryDeserializer] AdtReader::read_field(self : AdtReader, field_name : String, default : T?) -> T raise DesertFailure

    Read a required field.

    default supplies the value for fields that were added after the stored version; without a default such data fails with FieldWithoutDefaultValueIsMissing.

    AdtReader::read_optional_field

    source
    fn[T : BinaryDeserializer] AdtReader::read_optional_field(self : AdtReader, field_name : String, default : T??) -> T? raise DesertFailure

    Read an optional field, applying the FieldMadeOptional compatibility rules. default supplies the whole Option for fields added after the stored version.

    AdtWriter

    source
    pub struct AdtWriter {
      // private fields
    }

    Streaming writer for evolved records, the runtime engine behind derived codecs (mirrors desert-rust's AdtSerializer).

    For a record with V evolution steps (AdtMetadata::version):
    • new_v0 (V = 0) writes the 0 version byte and fields go straight to the active output — wire-identical to a plain tuple.
    • new (V > 0) writes the version byte, routes every field into the buffer of its generation chunk (write_field), and finish appends the evolution header followed by the chunk bytes.

    Fields not mentioned in any FieldAdded step belong to chunk 0.

    AdtWriter::finish

    source
    fn AdtWriter::finish(self : AdtWriter) -> Unit raise DesertFailure

    Append the evolution header and the chunk data (evolved records only; a no-op for version 0).

    AdtWriter::new

    source

    Create a writer for an evolved record: writes the version byte and prepares the per-generation chunk buffers.

    AdtWriter::new_v0

    source
    fn AdtWriter::new_v0(metadata : AdtMetadata, ctx : SerializationContext) -> AdtWriter raise DesertFailure

    Create a writer for a version-0 record: writes the version byte 0 and lets fields serialize directly to the active output.

    AdtWriter::write_constructor

    source
    fn AdtWriter::write_constructor(self : AdtWriter, constructor_idx : UInt, serialize_case : (SerializationContext) -> Unit raise DesertFailure) -> Unit raise DesertFailure

    Write a coproduct constructor index directly to the active output, followed by the case body produced by serialize_case.

    AdtWriter::write_field

    source
    fn[T : BinarySerializer] AdtWriter::write_field(self : AdtWriter, field_name : String, value : T) -> Unit raise DesertFailure

    Serialize one field of the record, routing it to its generation's chunk buffer (for evolved records).

    ArrayOfInt

    source
    pub struct ArrayOfInt {
      values : Array[Int]
    }

    Newtype marking an Array[Int] with the specialized codec (MoonBit coherence: a second BinarySerializer impl cannot be attached to Array[Int] itself).

    ArrayOfInt::new

    source
    fn ArrayOfInt::new(values : Array[Int]) -> ArrayOfInt

    Wrap an Array[Int] to use the specialized serializer. Deserialization via the plain Array[Int] codec accepts the identical bytes, so callers can keep decoding into a plain array.

    ArrayOfInt::serialize

    source
    fn ArrayOfInt::serialize(self : ArrayOfInt, ctx : SerializationContext) -> Unit raise DesertFailure

    BinaryInput

    source
    pub struct BinaryInput {
      // private fields
    }

    A byte source the deserializer reads from: an immutable Bytes with a mutable cursor. Reads are bounds-checked against the current region limit and fail with InputEndedUnexpectedly on underrun.

    The evolution machinery uses push_region/pop_region to fence reads to a byte range (an evolved record's generation chunk) and seek to jump between chunk start positions.

    BinaryInput::is_empty

    source
    fn BinaryInput::is_empty(self : BinaryInput) -> Bool

    True when the whole input has been consumed.

    BinaryInput::new

    source
    fn BinaryInput::new(data : Bytes) -> BinaryInput

    Create an input reading from data, starting at position 0.

    BinaryInput::pop_region

    source
    fn BinaryInput::pop_region(self : BinaryInput) -> (Int, Int)

    End the current region: restores the previous cursor state and returns the region's remaining (start, size) range so the caller can resume the same chunk later.

    BinaryInput::position

    source
    fn BinaryInput::position(self : BinaryInput) -> Int

    Number of bytes already consumed.

    BinaryInput::push_region

    source
    fn BinaryInput::push_region(self : BinaryInput, start : Int, size : Int) -> Unit

    Fence reads into [start, start + size): the cursor jumps to start, the region end becomes the read limit, and the previous cursor state is pushed onto the region stack.

    BinaryInput::read_bytes

    source
    fn BinaryInput::read_bytes(self : BinaryInput, count : Int) -> Bytes raise DesertFailure

    Read count raw bytes into an owned Bytes.

    BinaryInput::read_bytes_view

    source
    fn BinaryInput::read_bytes_view(self : BinaryInput, count : Int) -> BytesView raise DesertFailure

    Read count raw bytes as a zero-copy view.

    BinaryInput::read_f32

    source
    fn BinaryInput::read_f32(self : BinaryInput) -> Float raise DesertFailure

    Read a 4-byte IEEE-754 float (big-endian bit pattern).

    BinaryInput::read_f64

    source
    fn BinaryInput::read_f64(self : BinaryInput) -> Double raise DesertFailure

    Read an 8-byte IEEE-754 double (big-endian bit pattern).

    BinaryInput::read_i16

    source
    fn BinaryInput::read_i16(self : BinaryInput) -> Int raise DesertFailure

    Read a 2-byte big-endian signed integer, sign-extended into an Int.

    BinaryInput::read_i32

    source
    fn BinaryInput::read_i32(self : BinaryInput) -> Int raise DesertFailure

    Read a 4-byte big-endian signed integer.

    BinaryInput::read_i64

    source
    fn BinaryInput::read_i64(self : BinaryInput) -> Int64 raise DesertFailure

    Read an 8-byte big-endian signed integer.

    BinaryInput::read_i8

    source
    fn BinaryInput::read_i8(self : BinaryInput) -> Int raise DesertFailure

    Read a 1-byte signed integer, sign-extended into an Int.

    BinaryInput::read_u16

    source
    fn BinaryInput::read_u16(self : BinaryInput) -> UInt16 raise DesertFailure

    Read a 2-byte big-endian unsigned integer.

    BinaryInput::read_u32

    source
    fn BinaryInput::read_u32(self : BinaryInput) -> UInt raise DesertFailure

    Read a 4-byte big-endian unsigned integer.

    BinaryInput::read_u64

    source
    fn BinaryInput::read_u64(self : BinaryInput) -> UInt64 raise DesertFailure

    Read an 8-byte big-endian unsigned integer.

    BinaryInput::read_u8

    source
    fn BinaryInput::read_u8(self : BinaryInput) -> Byte raise DesertFailure

    Read a single raw byte.

    BinaryInput::read_var_i32

    source
    fn BinaryInput::read_var_i32(self : BinaryInput) -> Int raise DesertFailure

    Read a zigzag-mapped signed varint (symmetric with write_var_i32).

    BinaryInput::read_var_u32

    source
    fn BinaryInput::read_var_u32(self : BinaryInput) -> UInt raise DesertFailure

    Read a base-128 varint of an unsigned 32-bit value. Fails with DeserializationFailure if the encoding exceeds 5 bytes.

    BinaryInput::remaining

    source
    fn BinaryInput::remaining(self : BinaryInput) -> Int

    Number of bytes left to read.

    BinaryInput::reset_input

    source
    fn BinaryInput::reset_input(self : BinaryInput, data : Bytes) -> Unit

    Point the input at new data and clear any region state, reusing the allocated object. Session state lives in the DeserializationContext and is reset separately.

    BinaryInput::seek

    source
    fn BinaryInput::seek(self : BinaryInput, target : Int) -> Unit raise DesertFailure

    Move the cursor to an absolute position within the current region. Only used by the evolution machinery to jump to a chunk's start.

    BinaryInput::skip_bytes

    source
    fn BinaryInput::skip_bytes(self : BinaryInput, count : Int) -> Unit raise DesertFailure

    Advance the cursor by count bytes; negative counts fail cleanly instead of wrapping into a trap on a later view construction.

    ByteOutput

    source
    pub struct ByteOutput {
      // private fields
    }

    A BinaryOutput that accumulates into an in-memory growable byte buffer.

    Overrides the fixed-width operations with the bulk fast paths of the core Buffer.

    ByteOutput::length

    source
    fn ByteOutput::length(self : ByteOutput) -> Int

    Number of bytes written so far.

    ByteOutput::new

    source
    fn ByteOutput::new(capacity? : Int) -> ByteOutput

    Create an empty output. Pass capacity~ to pre-allocate.

    ByteOutput::reset

    source
    fn ByteOutput::reset(self : ByteOutput) -> Unit

    Discard the bytes written so far, reusing the allocated capacity.

    ByteOutput::to_bytes

    source
    fn ByteOutput::to_bytes(self : ByteOutput) -> Bytes

    The bytes written so far, as an immutable snapshot.

    ByteOutput::write_bytes

    source
    fn ByteOutput::write_bytes(self : ByteOutput, bytes : BytesView) -> Unit raise DesertFailure

    Promote the BinaryOutput operations to regular methods on ByteOutput.

    ByteOutput::write_f32

    source
    fn ByteOutput::write_f32(self : ByteOutput, value : Float) -> Unit raise DesertFailure

    Promote the BinaryOutput operations to regular methods on ByteOutput.

    ByteOutput::write_f64

    source
    fn ByteOutput::write_f64(self : ByteOutput, value : Double) -> Unit raise DesertFailure

    Promote the BinaryOutput operations to regular methods on ByteOutput.

    ByteOutput::write_i16

    source
    fn ByteOutput::write_i16(self : ByteOutput, value : Int) -> Unit raise DesertFailure

    Promote the BinaryOutput operations to regular methods on ByteOutput.

    ByteOutput::write_i32

    source
    fn ByteOutput::write_i32(self : ByteOutput, value : Int) -> Unit raise DesertFailure

    Promote the BinaryOutput operations to regular methods on ByteOutput.

    ByteOutput::write_i64

    source
    fn ByteOutput::write_i64(self : ByteOutput, value : Int64) -> Unit raise DesertFailure

    Promote the BinaryOutput operations to regular methods on ByteOutput.

    ByteOutput::write_i8

    source
    fn ByteOutput::write_i8(self : ByteOutput, value : Int) -> Unit raise DesertFailure

    Promote the BinaryOutput operations to regular methods on ByteOutput.

    ByteOutput::write_string_utf8

    source
    fn ByteOutput::write_string_utf8(self : ByteOutput, s : StringView) -> Unit raise DesertFailure

    Promote the BinaryOutput operations to regular methods on ByteOutput.

    ByteOutput::write_u16

    source
    fn ByteOutput::write_u16(self : ByteOutput, value : UInt16) -> Unit raise DesertFailure

    Promote the BinaryOutput operations to regular methods on ByteOutput.

    ByteOutput::write_u32

    source
    fn ByteOutput::write_u32(self : ByteOutput, value : UInt) -> Unit raise DesertFailure

    Promote the BinaryOutput operations to regular methods on ByteOutput.

    ByteOutput::write_u64

    source
    fn ByteOutput::write_u64(self : ByteOutput, value : UInt64) -> Unit raise DesertFailure

    Promote the BinaryOutput operations to regular methods on ByteOutput.

    ByteOutput::write_u8

    source
    fn ByteOutput::write_u8(self : ByteOutput, byte : Byte) -> Unit raise DesertFailure

    Promote the BinaryOutput operations to regular methods on ByteOutput.

    ByteOutput::write_var_i32

    source
    fn ByteOutput::write_var_i32(self : ByteOutput, value : Int) -> Unit raise DesertFailure

    Promote the BinaryOutput operations to regular methods on ByteOutput.

    ByteOutput::write_var_u32

    source
    fn ByteOutput::write_var_u32(self : ByteOutput, value : UInt) -> Unit raise DesertFailure

    Promote the BinaryOutput operations to regular methods on ByteOutput.

    DeduplicatedString

    source
    pub struct DeduplicatedString {
      // private fields
    } derive(Eq, 
    Debug    )

    Opt-in string deduplication wrapper (mirrors DeduplicatedString in both existing implementations).

    The first occurrence is written as a normal string and registered in the session's dedup table; every later occurrence is written as the zigzag-encoded negative id (var_i32(-id)). Ids start from 1. On the reading side every inline string is registered, so a negative value not present in the table fails with InvalidStringId.

    Do not mix deduplicated and plain string fields inside evolvable records: an older reader skipping a newly added deduplicated field would miss an id assignment and every later back-reference would be corrupted. Plain String fields are always evolution-safe.

    DeduplicatedString::deserialize

    source

    DeduplicatedString::equal

    source

    DeduplicatedString::get

    source
    fn DeduplicatedString::get(self : DeduplicatedString) -> String

    The wrapped string.

    DeduplicatedString::new

    source
    fn DeduplicatedString::new(string : String) -> DeduplicatedString

    DeduplicatedString::not_equal

    source

    DeduplicatedString::serialize

    source
    fn DeduplicatedString::serialize(self : DeduplicatedString, ctx : SerializationContext) -> Unit raise DesertFailure

    DeduplicatedString::to_repr

    source

    DeserializationContext

    source
    pub struct DeserializationContext {
      input : BinaryInput
      options : Options
      // private fields
    }

    Per-call context for deserialization: the input cursor, the session state and the format options. The state table is filled as strings are read.

    DeserializationContext::new

    source

    Construct a deserialization context over a custom input.

    DeserializationContext::options

    source

    The per-call format options.

    DeserializationContext::pop_region

    source
    fn DeserializationContext::pop_region(self : DeserializationContext) -> (Int, Int)

    End the current region, restoring the previous cursor state and returning the region's remaining unread range (the caller stores it to continue the chunk later).

    DeserializationContext::pos

    source

    Position of the input cursor.

    DeserializationContext::push_region

    source
    fn DeserializationContext::push_region(self : DeserializationContext, start : Int, size : Int) -> Unit

    Fence subsequent reads into region: the cursor jumps to the region's start and reads beyond its end fail. The previous cursor state is saved.

    DeserializationContext::read

    source

    Read a value of any BinaryDeserializer type from the context.

    DeserializationContext::read_bytes

    source
    fn DeserializationContext::read_bytes(self : DeserializationContext, count : Int) -> Bytes raise DesertFailure

    DeserializationContext::read_bytes_view

    source
    fn DeserializationContext::read_bytes_view(self : DeserializationContext, count : Int) -> BytesView raise DesertFailure

    DeserializationContext::read_compressed

    source
    fn DeserializationContext::read_compressed(self : DeserializationContext) -> Bytes raise DesertFailure

    Read a DEFLATE-compressed byte block.

    DeserializationContext::read_f32

    source
    fn DeserializationContext::read_f32(self : DeserializationContext) -> Float raise DesertFailure

    DeserializationContext::read_f64

    source
    fn DeserializationContext::read_f64(self : DeserializationContext) -> Double raise DesertFailure

    DeserializationContext::read_flexible_array

    source

    Deserialize an array accepting both wire forms of the generic collection encoding:
    • counted: var_i32(n) followed by n element payloads
    • unknown-size: var_i32(-1), then 0x01 + element for each item, terminated by 0x00

    DeserializationContext::read_i16

    source

    DeserializationContext::read_i32

    source

    DeserializationContext::read_i64

    source
    fn DeserializationContext::read_i64(self : DeserializationContext) -> Int64 raise DesertFailure

    DeserializationContext::read_i8

    source

    DeserializationContext::read_optional_value

    source
    fn[T : BinaryDeserializer] DeserializationContext::read_optional_value(self : DeserializationContext) -> T? raise DesertFailure

    Read an Option[T] payload (tag byte + optional value). Helper for the evolution machinery and derived codecs.

    DeserializationContext::read_string_payload

    source
    fn DeserializationContext::read_string_payload(self : DeserializationContext) -> String raise DesertFailure

    Read a UTF-8 encoded string payload (symmetric with write_string_payload).

    DeserializationContext::read_u16

    source
    fn DeserializationContext::read_u16(self : DeserializationContext) -> UInt16 raise DesertFailure

    DeserializationContext::read_u32

    source

    DeserializationContext::read_u64

    source
    fn DeserializationContext::read_u64(self : DeserializationContext) -> UInt64 raise DesertFailure

    DeserializationContext::read_u8

    source

    DeserializationContext::read_unknown

    source

    Read a registry-dispatched value: reads the var_u32 id, looks the entry up and calls its typed constructor. Unknown ids fail with InvalidTypeId — there is no way to skip an unknown payload, since the format carries no length information for it.

    DeserializationContext::read_var_i32

    source
    fn DeserializationContext::read_var_i32(self : DeserializationContext) -> Int raise DesertFailure

    DeserializationContext::read_var_u32

    source
    fn DeserializationContext::read_var_u32(self : DeserializationContext) -> UInt raise DesertFailure

    DeserializationContext::reset_input

    source
    fn DeserializationContext::reset_input(self : DeserializationContext, data : Bytes) -> Unit

    Point a deserialization context at new data: resets the input cursor and installs a fresh session state (dedup ids never leak across calls). Lets hot loops reuse the context instead of paying per-call allocation.

    DeserializationContext::skip

    source
    fn DeserializationContext::skip(self : DeserializationContext, count : Int) -> Unit raise DesertFailure

    Advance the input cursor by count bytes (no data is returned).

    Evolution

    source
    pub(all) enum Evolution {
      InitialVersion
      FieldAdded(String)
      FieldMadeOptional(String)
      FieldRemoved(String)
      FieldMadeTransient(String)
    }

    Schema evolution steps, mirroring the Scala Evolution sealed trait and the Rust Evolution enum.

    A record's evolution history is declared as InitialVersion followed by one step per schema change; the list length determines the wire version byte (steps.length() - 1).

    Options

    source
    pub struct Options {
      chars_as_u16 : Bool
    } derive(Default, 
    Debug    )

    Per-call serialization options.

    These flags only change the wire format when set; both sides of a stream must agree on them (same contract as desert-rust's Options).

    Options::default

    source
    fn Options::default() -> Options

    Options::scala_compatible

    source
    fn Options::scala_compatible() -> Options

    Options matching the original Scala library's wire behavior.

    Options::to_repr

    source

    RegisteredBox

    source
    pub struct RegisteredBox {
      id : Int
      type_name : String
      value : &RegisteredValue
    }

    A deserialized unknown value: the resolved registry id and entry name, plus the boxed value. Extract the concrete value by matching on type_name (MoonBit has no runtime downcast; model open sets of known types as an enum of yours and register the enum).

    RegisteredEntry

    source
    pub struct RegisteredEntry {
      id : Int
      type_name : String
      deserialize : (DeserializationContext) -> &RegisteredValue raise DesertFailure
    }

    One registered type: its id, name and typed deserialization constructor.

    SerializationContext

    source
    pub struct SerializationContext {
      state : SerializerState
      options : Options
      // private fields
    }

    Per-call context for serialization: the output sink, the session state (string dedup / reference tracking) and the format options.

    The active output can be swapped by the evolution machinery (push_output/pop_output) so that a record's fields are captured into per-generation chunk buffers.

    SerializationContext::new

    source

    Construct a serialization context over a custom output sink.

    SerializationContext::options

    source

    The per-call format options.

    SerializationContext::pop_output

    source

    Restore the previous output sink, returning the one that was active.

    SerializationContext::push_output

    source
    fn SerializationContext::push_output(self : SerializationContext, output : &BinaryOutput) -> Unit

    Reroute subsequent writes into output, saving the previous sink on the context's stack (used by the evolution machinery to capture a record's fields into per-generation chunk buffers).

    SerializationContext::write

    source
    fn[T : BinarySerializer] SerializationContext::write(self : SerializationContext, value : T) -> Unit raise DesertFailure

    Write a value of any BinarySerializer type into the context.

    SerializationContext::write_bytes

    source
    fn SerializationContext::write_bytes(self : SerializationContext, bytes : BytesView) -> Unit raise DesertFailure

    Promote the BinaryOutput operations to regular methods, so codecs can write ctx.write_u8(...) fluently.

    SerializationContext::write_compressed

    source
    fn SerializationContext::write_compressed(self : SerializationContext, data : Bytes) -> Unit raise DesertFailure

    Write data DEFLATE-compressed into the output.

    SerializationContext::write_f32

    source
    fn SerializationContext::write_f32(self : SerializationContext, value : Float) -> Unit raise DesertFailure

    Promote the BinaryOutput operations to regular methods, so codecs can write ctx.write_u8(...) fluently.

    SerializationContext::write_f64

    source
    fn SerializationContext::write_f64(self : SerializationContext, value : Double) -> Unit raise DesertFailure

    Promote the BinaryOutput operations to regular methods, so codecs can write ctx.write_u8(...) fluently.

    SerializationContext::write_i16

    source
    fn SerializationContext::write_i16(self : SerializationContext, value : Int) -> Unit raise DesertFailure

    Promote the BinaryOutput operations to regular methods, so codecs can write ctx.write_u8(...) fluently.

    SerializationContext::write_i32

    source
    fn SerializationContext::write_i32(self : SerializationContext, value : Int) -> Unit raise DesertFailure

    Promote the BinaryOutput operations to regular methods, so codecs can write ctx.write_u8(...) fluently.

    SerializationContext::write_i64

    source
    fn SerializationContext::write_i64(self : SerializationContext, value : Int64) -> Unit raise DesertFailure

    Promote the BinaryOutput operations to regular methods, so codecs can write ctx.write_u8(...) fluently.

    SerializationContext::write_i8

    source
    fn SerializationContext::write_i8(self : SerializationContext, value : Int) -> Unit raise DesertFailure

    Promote the BinaryOutput operations to regular methods, so codecs can write ctx.write_u8(...) fluently.

    SerializationContext::write_iter_unknown_size

    source
    fn[T : BinarySerializer] SerializationContext::write_iter_unknown_size(self : SerializationContext, iter : Iter[T]) -> Unit raise DesertFailure

    Serialize a lazy iterator using the unknown-size collection form (var_i32(-1), then 0x01 + element per item, terminated by 0x00).

    Use this when the number of items is not known upfront (streaming); when the size is known, prefer a concrete collection codec, which is more compact (no per-element tag).

    SerializationContext::write_string_payload

    source
    fn SerializationContext::write_string_payload(self : SerializationContext, s : String) -> Unit raise DesertFailure

    Write a UTF-8 encoded string payload: var_i32(byte_length) followed by the UTF-8 bytes. This is the primitive used by the String codec and by the evolution machinery for field names.

    SerializationContext::write_string_utf8

    source
    fn SerializationContext::write_string_utf8(self : SerializationContext, s : StringView) -> Unit raise DesertFailure

    Promote the BinaryOutput operations to regular methods, so codecs can write ctx.write_u8(...) fluently.

    SerializationContext::write_u16

    source
    fn SerializationContext::write_u16(self : SerializationContext, value : UInt16) -> Unit raise DesertFailure

    Promote the BinaryOutput operations to regular methods, so codecs can write ctx.write_u8(...) fluently.

    SerializationContext::write_u32

    source
    fn SerializationContext::write_u32(self : SerializationContext, value : UInt) -> Unit raise DesertFailure

    Promote the BinaryOutput operations to regular methods, so codecs can write ctx.write_u8(...) fluently.

    SerializationContext::write_u64

    source
    fn SerializationContext::write_u64(self : SerializationContext, value : UInt64) -> Unit raise DesertFailure

    Promote the BinaryOutput operations to regular methods, so codecs can write ctx.write_u8(...) fluently.

    SerializationContext::write_u8

    source
    fn SerializationContext::write_u8(self : SerializationContext, byte : Byte) -> Unit raise DesertFailure

    Promote the BinaryOutput operations to regular methods, so codecs can write ctx.write_u8(...) fluently.

    SerializationContext::write_unknown

    source
    fn SerializationContext::write_unknown(self : SerializationContext, registry : TypeRegistry, value : &RegisteredValue) -> Unit raise DesertFailure

    Serialize a value through the registry: writes var_u32(type_id) followed by the registered codec's payload, inline — no markers, no length prefix (wire-compatible with both original implementations).

    Fails with TypeNotRegistered when the value's type name has no id.

    SerializationContext::write_var_i32

    source
    fn SerializationContext::write_var_i32(self : SerializationContext, value : Int) -> Unit raise DesertFailure

    Promote the BinaryOutput operations to regular methods, so codecs can write ctx.write_u8(...) fluently.

    SerializationContext::write_var_u32

    source
    fn SerializationContext::write_var_u32(self : SerializationContext, value : UInt) -> Unit raise DesertFailure

    Promote the BinaryOutput operations to regular methods, so codecs can write ctx.write_u8(...) fluently.

    SerializerState

    source
    pub struct SerializerState {
      // private fields
    }

    Mutable per-session state shared by serialization and deserialization: the deduplicated-string table and (reserved for later phases) the object reference table. A fresh state is created for every top-level serialize/deserialize call, so ids never leak across calls.

    The tables are allocated lazily on first use, matching desert-rust: sessions that never touch deduplication (the common case for small payloads) pay nothing for them.

    SerializerState::new

    source

    TypeRegistry

    source
    pub struct TypeRegistry {
      // private fields
    }

    An immutable type registry: maps registry ids to typed constructors and type names to ids.

    TypeRegistry::empty

    source
    fn TypeRegistry::empty() -> TypeRegistry

    A registry where every lookup fails (equivalent of TypeRegistry.empty).

    TypeRegistry::entry_of

    source
    fn TypeRegistry::entry_of(self : TypeRegistry, id : Int) -> RegisteredEntry?

    Look up the entry for a registry id.

    TypeRegistry::id_of

    source
    fn TypeRegistry::id_of(self : TypeRegistry, type_name : String) -> Int?

    Look up the id registered for a type name.

    TypeRegistryBuilder

    source
    pub struct TypeRegistryBuilder {
      // private fields
    }

    Builder API assigning consecutive ids starting from 1, mirroring the Scala DefaultTypeRegistry:

    let registry = TypeRegistryBuilder::new()
      .register::<MyTypeA>(fn(ctx) { MyTypeA::deserialize(ctx) })
      .register_placeholder()
      .register::<MyTypeB>(fn(ctx) { MyTypeB::deserialize(ctx) })
      .freeze()

    register_placeholder reserves an id slot without a type, keeping ids stable across releases when a type is retired. Lookups are exact type-name matches (MoonBit has no runtime class hierarchy test).

    TypeRegistryBuilder::freeze

    source

    TypeRegistryBuilder::new

    source

    TypeRegistryBuilder::register

    source
    fn TypeRegistryBuilder::register(self : TypeRegistryBuilder, type_name : String, deserialize : (DeserializationContext) -> &RegisteredValue raise DesertFailure) -> TypeRegistryBuilder raise DesertFailure

    Register the next id for deserialize, a typed constructor of a RegisteredValue implementation. Fails if the type name is already registered.

    TypeRegistryBuilder::register_placeholder

    source
    fn TypeRegistryBuilder::register_placeholder(self : TypeRegistryBuilder) -> TypeRegistryBuilder

    Reserve the next id without registering a type.

    compress_bytes

    source
    fn compress_bytes(data : Bytes) -> Bytes raise DesertFailure

    Round-trip helpers for testing and ad-hoc use.

    decompress_bytes

    source
    fn decompress_bytes(data : Bytes) -> Bytes raise DesertFailure

    deserialize

    source
    fn[T : BinaryDeserializer] deserialize(data : Bytes, options? : Options) -> T raise DesertFailure

    Deserialize a value of type T from data.

    The input must have been produced by a compatible serializer (same options, same schema version). Underruns and format violations raise DesertFailure.

    deserialize_exhaustive

    source
    fn[T : BinaryDeserializer] deserialize_exhaustive(data : Bytes, options? : Options) -> T raise DesertFailure

    Deserialize a value of type T from data, additionally checking that the whole input was consumed. Trailing bytes fail with DeserializationFailure, which catches framing mistakes early.

    deserialize_unknown_from_bytes

    source
    fn deserialize_unknown_from_bytes(data : Bytes, registry : TypeRegistry, options? : Options) -> RegisteredBox raise DesertFailure

    Top-level convenience: read a registry-dispatched value.

    read_utf8

    source
    fn read_utf8(bytes : BytesView) -> String raise DesertFailure

    Decode UTF-8 bytes into a MoonBit (UTF-16) string.

    Malformed sequences fail with DeserializationFailure, matching the Rust library (String::from_utf8). All standard UTF-8 forms are accepted, including 4-byte sequences for astral code points.

    serialize_to_bytes

    source
    fn[T : BinarySerializer] serialize_to_bytes(value : T, options? : Options) -> Bytes raise DesertFailure

    Serialize value to an in-memory byte buffer, returning the bytes.

    A fresh session state (string dedup table, reference table) is created for the call, matching the top-level APIs of both existing implementations: ids never leak across calls.

    serialize_to_output

    source
    fn[T : BinarySerializer] serialize_to_output(value : T, out : ByteOutput, options? : Options) -> Unit raise DesertFailure

    Serialize value into a caller-provided ByteOutput.

    serialize_unknown_to_bytes

    source
    fn[T : RegisteredValue] serialize_unknown_to_bytes(value : T, registry : TypeRegistry, options? : Options) -> Bytes raise DesertFailure

    Top-level convenience: serialize an erased value to bytes.

    serialized_size

    source
    fn[T : BinarySerializer] serialized_size(value : T, options? : Options) -> Int raise DesertFailure

    Compute the serialized size of value in bytes (the bytes are written to a scratch output and discarded). Two-pass exact-size serialization can be built on this.

    utf8_byte_length

    source
    fn utf8_byte_length(s : String) -> Int

    Number of UTF-8 bytes needed to encode s.

    MoonBit strings are UTF-16 internally; lone surrogates are counted as one byte because they are encoded as the ASCII ? substitution character (matching Scala's String.getBytes(UTF_8) REPLACE behavior).

    write_utf8

    source
    fn write_utf8(s : String, out : FixedArray[Byte], offset : Int) -> Int

    Encode s as UTF-8 into out, which must have space for at least utf8_byte_length(s) bytes. Returns the number of bytes written.

    Lone surrogates are substituted with the ASCII ? character (0x3F), matching Scala's String.getBytes(UTF_8) replacement behavior.

    Source Files