🥮 mooncakes.io

    moonsbe

    SBE binary message codecs and MoonBit code generation with reference interoperability tests

    sbe
    binary
    codec
    codegen
    interoperability
    Download zip
    Author
    Version
    0.2.0
    License
    Apache-2.0
    Repository
    Last updated
    1 hour ago
    Downloads
    14

    #moonsbe

    CI

    MoonBit 的 SBE 二进制消息编解码与类型化代码生成库。输入现有 SBE XML Schema，生成 MoonBit 结构体、枚举和编解码适配器，用于与 Java、C++ 等系统交换同一消息定义下的二进制数据。

    0.2.0 支持自定义消息头、扩展组维度、版本化复合成员和可选复合字段。Java 与 MoonBit 已对照 207 字节的官方 Car 报文及两条 36 字节扩展报文，完整报文逐字节一致。可用 moon add liyun6666/moonsbe@0.2.0 安装。Mooncakes 包页面

    #快速运行

    需要 MoonBit、Node.js 22+。当前验证工具链为 moon 0.1.20261023、moonc v0.10.17。Java 互通验证另需 JDK 24。首次解析依赖及下载参考工具需要联网。

    moon update
moon test --target js --warn-list=-implicit_impl_as_method --deny-warn
moon run cmd/main --target js --warn-list=-implicit_impl_as_method -- schema interop/upstream/example-schema.xml
moon run cmd/main --target js --warn-list=-implicit_impl_as_method -- encode interop/upstream/example-schema.xml 1 examples/car.json car.bin
moon run cmd/main --target js --warn-list=-implicit_impl_as_method -- decode interop/upstream/example-schema.xml car.bin

    参考 Schema 保留其 common-types.xml XInclude。库调用方显式提供资源内容，CLI 在 Schema 所在目录读取受限的相对路径。

    #类型化代码生成

    moon run cmd/main --target js --warn-list=-implicit_impl_as_method -- generate interop/upstream/example-schema.xml generated/car/messages.mbt
moon fmt generated/car
moon check --target js --warn-list=-implicit_impl_as_method --deny-warn

    消费包在 moon.pkg 中导入生成包。

    import {
  "liyun6666/moonsbe/generated/car" @car,
}

    生成的 Car 提供 Car::decode(bytes) 和 car.encode()。f_serial_number 是 UInt64，f_available 是生成的枚举，重复组为类型化记录数组。examples/typed 使用这些接口读取字段并重新编码，其输出由 Java 参考解码器核验。

    moon run tools/interop.mbtx

    该命令校验参考 JAR 的 SHA-256，执行独立编码、全部字段对照、完整字节比较、生成代码编译和消费包验证。日志位于 _build/interop-logs，CI 将日志与二进制文件归档。

    只需读取根块的定长数值字段时，可以直接使用生成的定长视图。

    let view = @car.Car::view(bytes)
    let serial_number = view.f_serial_number()
    let first_number = view.f_some_numbers(0)

    视图按生成时确定的偏移、宽度和字节序从原始报文读数，并检查版本和访问边界。完整 encode 与 decode 在首次调用时加载 Schema，使用共享运行时处理复合字段、重复组和变长区。复现单字段读取的测量见 互通记录。

    #库接口

    根包只依赖 MoonBit core，文件读写位于 CLI 和示例包。主要接口包括以下内容。

    接口用途
    parse_schema(xml, includes=...)编译 XML Schema，解析类型、布局和显式提供的 XInclude
    value_from_json(schema, template_id, source)根据 Schema 读取消息 JSON，保留整数精度并拒绝重复键
    encode(schema, template_id, value, version=...)按 Schema 指定的头部布局编码一条完整消息
    decode(schema, bytes, max_entries=...)解码一条已有边界的完整消息，检查长度与资源上限
    generate_moonbit(xml, includes=...)生成类型化适配器与定长数值字段视图
    compare_schemas(old, newer)检查此运行时下的双向版本兼容风险

    Value 区分 Signed(Int64)、Unsigned(UInt64)、Floating(Double)、Text、Binary、Record、Sequence 和 Absent。JSON 输入可使用精确整数文本或十进制字符串，输出整数统一采用字符串，二进制值采用十六进制字符串。

    Schema 的集合公开用于检查与代码生成，调用方应将编译结果视为只读。校验错误使用 SbeError，包含 Schema 位置、字段原因或消息偏移信息。

    #CLI

    以下命令加上前缀 moon run cmd/main --target js --warn-list=-implicit_impl_as_method --。

    schema SCHEMA.xml
encode SCHEMA.xml TEMPLATE_ID MESSAGE.json OUTPUT.bin
decode SCHEMA.xml INPUT.bin [OUTPUT.json]
generate SCHEMA.xml OUTPUT.mbt
compare OLD_SCHEMA.xml NEW_SCHEMA.xml
hex INPUT.bin

    校验或文件操作失败返回退出码 2。Schema 比较发现兼容性风险返回 3。消息边界由调用方的传输层或文件协议提供，每次 decode 处理一条完整消息。

    #验证与代码统计

    moon fmt --check
moon check --target js --warn-list=-implicit_impl_as_method --deny-warn
moon test --target js --warn-list=-implicit_impl_as_method --deny-warn
moon test --target wasm-gc -p liyun6666/moonsbe -p liyun6666/moonsbe/examples/extended --warn-list=-implicit_impl_as_method --deny-warn
moon run tools/interop.mbtx
moon run tools/stats.mbtx

    统计脚本需要 ripgrep，将手写实现与测试分开，排除生成代码、上游 Java/XML、接口文件、工具脚本、文档与构建结果。测试覆盖截断、越界、整数范围、空值、名称冲突、嵌套组、字符编码和版本变化。implicit_impl_as_method 为当前 MoonBit 派生方法迁移告警，其余告警作为错误处理。

    #来源与许可证

    MoonBit 实现按 SBE 公开规范编写，项目采用 Apache License 2.0。互通所需的上游 Schema 与示例 Java 源码来自 aeron-io/simple-binary-encoding 的 1.35.0 标签，保留原始内容与版权声明。引用材料和生成产物按独立类别统计。详情见 LICENSE、NOTICE 和 interop/UPSTREAM.md。

    SbeError

    source
    pub suberror SbeError {
      InvalidSchema(String, Int)
      Unsupported(String)
      Truncated(Int, Int, Int)
      InvalidValue(String, String)
      InvalidMessage(String)
      LimitExceeded(String)
    } derive(
    Debug    )

    Structured errors include an XML position or byte offset when available.

    BlockDef

    source
    pub(all) struct BlockDef {
      fields : Array[FieldDef]
      groups : Array[GroupDef]
      data : Array[DataDef]
      block_length : Int
    } derive(
    Debug    )

    ByteOrder

    source
    pub(all) enum ByteOrder {
      LittleEndian
      BigEndian
    } derive(Eq, 
    Debug    )

    CompositeMember

    source
    pub(all) struct CompositeMember {
      name : String
      type_name : String
      offset : Int
      since_version : Int
    } derive(
    Debug    )

    DataDef

    source
    pub(all) struct DataDef {
      name : String
      id : Int
      since_version : Int
      length_primitive : Primitive
      max_length : Int
      character_encoding : String
    } derive(
    Debug    )

    DecodedMessage

    source
    pub(all) struct DecodedMessage {
      header : MessageHeader
      value : Value
    } derive(
    Debug    )

    EnumCase

    source
    pub(all) struct EnumCase {
      name : String
      value : String
      since_version : Int
    } derive(
    Debug    )

    EvolutionReport

    source
    pub(all) struct EvolutionReport {
      old_version : Int
      new_version : Int
      old_reads_new : Bool
      new_reads_old : Bool
      differences : Array[SchemaDifference]
    } derive(
    Debug    , 
    ToJson    )

    Conservative compatibility checks for this runtime's strict decoding profile. An empty report is not a proof about arbitrary producer values.

    FieldDef

    source
    pub(all) struct FieldDef {
      name : String
      id : Int
      type_name : String
      offset : Int
      since_version : Int
      presence : Presence
      value_ref : String?
    } derive(
    Debug    )

    FixedBlock

    source
    pub struct FixedBlock {
      bytes : Bytes
      start : Int
      length : Int
      order : ByteOrder
    }

    A read-only window over the original bytes. Used by generated fixed-field accessors to avoid copying the block or materializing a message record.

    FixedBlock::float_at

    source
    fn FixedBlock::float_at(self : FixedBlock, offset : Int, width : Int) -> Double raise SbeError

    FixedBlock::int_at

    source
    fn FixedBlock::int_at(self : FixedBlock, offset : Int, width : Int) -> Int64 raise SbeError

    FixedBlock::new

    source
    fn FixedBlock::new(bytes : Bytes, start : Int, length : Int, order : ByteOrder) -> FixedBlock raise SbeError

    FixedBlock::uint_at

    source
    fn FixedBlock::uint_at(self : FixedBlock, offset : Int, width : Int) -> UInt64 raise SbeError

    FrameLayout

    source
    pub(all) struct FrameLayout {
      size : Int
      members : Array[FrameMember]
    } derive(Eq, 
    Debug    )

    The compiled wire layout of a message header or a group dimension.

    FrameMember

    source
    pub(all) struct FrameMember {
      name : String
      offset : Int
      primitive : Primitive
      min_value : Int
      max_value : Int
    } derive(Eq, 
    Debug    )

    A required unsigned scalar at its schema-defined position in a header.

    GroupDef

    source
    pub(all) struct GroupDef {
      name : String
      id : Int
      since_version : Int
      block : BlockDef
      count_primitive : Primitive
      max_count : Int
      dimension : FrameLayout
    } derive(
    Debug    )

    MessageDef

    source
    pub(all) struct MessageDef {
      name : String
      id : Int
      since_version : Int
      block : BlockDef
    } derive(
    Debug    )

    MessageHeader

    source
    pub(all) struct MessageHeader {
      block_length : Int
      template_id : Int
      schema_id : Int
      version : Int
    } derive(Eq, 
    Debug    )

    The standard eight-byte SBE message header, without transport framing.

    MessageHeader::read

    source
    fn MessageHeader::read(reader : Reader) -> MessageHeader raise SbeError

    MessageHeader::write

    source
    fn MessageHeader::write(self : MessageHeader, writer : Writer) -> Unit raise SbeError

    Presence

    source
    pub(all) enum Presence {
      Required
      Optional
      Constant
    } derive(Eq, 
    Debug    )

    Primitive

    source
    pub(all) enum Primitive {
      Char
      Int8
      UInt8
      Int16
      UInt16
      Int32
      UInt32
      Int64
      UInt64
      Float32
      Float64
    } derive(Eq, 
    Debug    )

    Primitive::is_float

    source
    fn Primitive::is_float(self : Primitive) -> Bool

    Primitive::is_signed

    source
    fn Primitive::is_signed(self : Primitive) -> Bool

    Primitive::parse

    source
    fn Primitive::parse(name : String) -> Primitive raise SbeError

    Primitive::width

    source
    fn Primitive::width(self : Primitive) -> Int

    Reader

    source
    pub struct Reader {
      bytes : Bytes
      order : ByteOrder
      position : Int
    }

    Bounded binary input. Cursor reads never silently clamp a truncated frame.

    Reader::new

    source
    fn Reader::new(bytes : Bytes, order? : ByteOrder) -> Reader

    Reader::position

    source
    fn Reader::position(self : Reader) -> Int

    Reader::read_bytes

    source
    fn Reader::read_bytes(self : Reader, length : Int) -> Bytes raise SbeError

    Reader::read_float

    source
    fn Reader::read_float(self : Reader, width : Int) -> Double raise SbeError

    Reader::read_int

    source
    fn Reader::read_int(self : Reader, width : Int) -> Int64 raise SbeError

    Reader::read_uint

    source
    fn Reader::read_uint(self : Reader, width : Int) -> UInt64 raise SbeError

    Read an unsigned word of 1, 2, 4 or 8 bytes without loss of precision.

    Reader::remaining

    source
    fn Reader::remaining(self : Reader) -> Int

    Reader::seek

    source
    fn Reader::seek(self : Reader, position : Int) -> Unit raise SbeError

    Schema

    source
    pub(all) struct Schema {
      id : Int
      version : Int
      package_name : String
      byte_order : ByteOrder
      header_layout : FrameLayout
      types : Map[String, TypeDef]
      messages : Array[MessageDef]
    } derive(
    Debug    )

    A compiled schema. Treat collections as immutable after compilation.

    Schema::encoded_size

    source
    fn Schema::encoded_size(self : Schema, name : String) -> Int raise SbeError

    Schema::message

    source
    fn Schema::message(self : Schema, id : Int) -> MessageDef raise SbeError

    Schema::type_def

    source
    fn Schema::type_def(self : Schema, name : String) -> TypeDef raise SbeError

    SchemaDifference

    source
    pub(all) struct SchemaDifference {
      direction : String
      code : String
      path : String
      detail : String
    } derive(
    Debug    , 
    ToJson    )

    TypeDef

    source
    pub(all) enum TypeDef {
      Encoded(TypeEncoding)
      Enumeration(Primitive, Array[EnumCase], Presence)
      BitSet(Primitive, Array[EnumCase])
      Composite(Array[CompositeMember], Int)
    } derive(
    Debug    )

    TypeEncoding

    source
    pub(all) struct TypeEncoding {
      primitive : Primitive
      length : Int
      presence : Presence
      character_encoding : String
      min_value : String?
      max_value : String?
      null_value : String?
      constant : String?
    } derive(
    Debug    )

    Value

    source
    pub(all) enum Value {
      Signed(Int64)
      Unsigned(UInt64)
      Floating(Double)
      Text(String)
      Binary(Bytes)
      Record(Map[String, Value])
      Sequence(Array[Value])
      Absent
    } derive(
    Debug    )

    Typed values retain full uint64 and int64 precision on every backend. Integers are strings in JSON; binary values use lowercase hexadecimal.

    Value::as_binary

    source
    fn Value::as_binary(self : Value) -> Bytes raise SbeError

    Value::as_float

    source
    fn Value::as_float(self : Value) -> Double raise SbeError

    Value::as_record

    source
    fn Value::as_record(self : Value) -> Map[String, Value] raise SbeError

    Value::as_sequence

    source
    fn Value::as_sequence(self : Value) -> Array[Value] raise SbeError

    Value::as_signed

    source
    fn Value::as_signed(self : Value) -> Int64 raise SbeError

    Value::as_text

    source
    fn Value::as_text(self : Value) -> String raise SbeError

    Value::as_unsigned

    source
    fn Value::as_unsigned(self : Value) -> UInt64 raise SbeError

    Value::field

    source
    fn Value::field(self : Value, name : String) -> Value raise SbeError

    Value::to_json

    source

    Writer

    source
    pub struct Writer {
      bytes : Array[Byte]
      order : ByteOrder
      limit : Int
    }

    A growable writer with an explicit output bound; padding is always zero.

    Writer::new

    source
    fn Writer::new(order? : ByteOrder, limit? : Int) -> Writer

    Writer::pad_to

    source
    fn Writer::pad_to(self : Writer, offset : Int) -> Unit raise SbeError

    Writer::position

    source
    fn Writer::position(self : Writer) -> Int

    Writer::to_bytes

    source
    fn Writer::to_bytes(self : Writer) -> Bytes

    Writer::write_bytes

    source
    fn Writer::write_bytes(self : Writer, bytes : Bytes) -> Unit raise SbeError

    Writer::write_float

    source
    fn Writer::write_float(self : Writer, value : Double, width : Int) -> Unit raise SbeError

    Writer::write_int

    source
    fn Writer::write_int(self : Writer, value : Int64, width : Int) -> Unit raise SbeError

    Writer::write_uint

    source
    fn Writer::write_uint(self : Writer, value : UInt64, width : Int) -> Unit raise SbeError

    compare_schemas

    source
    fn compare_schemas(old : Schema, newer : Schema) -> EvolutionReport raise SbeError

    Compare two compiled schema versions. Results describe this library's strict wire and symbolic-value profile, not general SBE conformance.

    decode

    source
    fn decode(schema : Schema, bytes : Bytes, max_entries? : Int) -> DecodedMessage raise SbeError

    Decode exactly one transport-delimited message. Extra trailing bytes are rejected rather than interpreted as another frame or silently discarded.

    encode

    source
    fn encode(schema : Schema, template_id : Int, value : Value, version? : Int, max_bytes? : Int) -> Bytes raise SbeError

    Encode one complete SBE message including its schema-defined header.

    from_hex

    source
    fn from_hex(text : String) -> Bytes raise SbeError

    generate_moonbit

    source
    fn generate_moonbit(xml : String, includes? : Map[String, String]) -> String raise SbeError

    Generate typed MoonBit adapters and direct-access fixed numeric views. Whole-message adapters use the shared checked runtime. The result imports this module under the alias sbe.

    parse_schema

    source
    fn parse_schema(xml : String, includes? : Map[String, String]) -> Schema raise SbeError

    Compile an SBE XML schema. Include resources are supplied explicitly by name; the compiler does not access the network or local filesystem.

    schema_include_names

    source
    fn schema_include_names(source : String) -> Array[String] raise SbeError

    Return directly requested include names, allowing a host to supply bytes.

    to_hex

    source
    fn to_hex(bytes : Bytes) -> String

    value_error

    source
    fn value_error(path : String, reason : String) -> SbeError

    Public factory used by generated adapters when rejecting a typed value.

    value_from_json

    source
    fn value_from_json(schema : Schema, template_id : Int, source : String) -> Value raise SbeError

    Parse a message JSON document without losing integer precision. Duplicate keys are rejected before any schema-driven conversion occurs.

    Source Files