pbf

    Protocol Buffers wire format in MoonBit: a zero-copy, allocation-free reader and a writer

    protobuf
    pbf
    decoder
    encoder
    binary
    Download zip
    Author
    Version
    0.1.0
    License
    Apache-2.0
    Last updated
    5 hours ago
    Downloads
    5

    No dependencies

    #pbf

    The Protocol Buffers wire format in MoonBit: a zero-copy reader that walks messages by hand, and a small writer. Pure MoonBit on every backend (wasm, wasm-gc, js, native), with no dependencies outside the standard library. There is no .proto compiler: you read the tags and call the reader for each field, as pbf users do in JavaScript.

    Part of mbt-map, a WebGIS library, where it reads vector tiles and ArcGIS feature responses; published on its own.

    moon add t-ujiie-g/pbf

    /// The `name` (field 1, a string) and the sum of `sizes` (field 2, packed varints) of a message.
    fn read(bytes : Bytes) -> (String, Int)? {
    let r = @pbf.Reader::new(bytes[:])
    let mut name = ""
    let mut sum = 0
    while !r.at_end() {
    let tag = r.read_tag()
    if tag == @pbf.tag(1, @pbf.WIRE_LEN) {
    name = r.read_string()
    } else if tag == @pbf.tag(2, @pbf.WIRE_LEN) {
    let token = r.enter()
    while !r.at_end() {
    sum += r.read_varint32()
    }
    r.exit(token)
    } else {
    r.skip(tag)
    }
    }
    guard r.error() is None else { return None }
    Some((name, sum))
    }

    #What it gives

    • Reader: a cursor over a BytesView. read_tag, read_varint32 / read_varint64, read_svarint32 / read_sint64 (zigzag), read_int64, read_bool, read_fixed32 / read_fixed64, read_float / read_double, read_bytes (a view of the buffer, no copy), read_string (UTF-8, invalid sequences become U+FFFD as TextDecoder does), skip(tag).
    • Nested messages and packed fields without allocating: enter() narrows the reader to the next length-delimited field and exit(token) leaves it; read_len_start() skips a field and gives where it starts, and reset(start, end) moves a reader back over a range.
    • Errors are sticky, never raised: a failed read records the first PbfError (UnexpectedEof, VarintTooLong, UnsupportedWireType, LengthOutOfRange, each with its byte offset), returns 0 and moves to the end, so at_end loops stop. Check error() or failed() once per message. In wasm-gc a call to a function that may raise allocates even when it succeeds, which is why the hot path has none.
    • Writer: write_tag, write_varint, write_varint32, write_svarint32, write_sint64, write_fixed32, write_fixed64, write_bytes, write_string, to_bytes. It is a minimal encoder meant for test data and small messages.
    • tag(field, wire) and WIRE_VARINT, WIRE_FIXED64, WIRE_LEN, WIRE_FIXED32.

    The input is treated as untrusted: every read is bounds-checked, varints longer than 10 bytes and lengths past the end of the range are errors, and groups (wire types 3 and 4) are refused.

    #License

    Apache-2.0. The module is written from the Protocol Buffers encoding documentation and contains no third-party code (see NOTICE).

    PbfError

    pub(all) suberror PbfError {
    UnexpectedEof(pos~ : Int)
    VarintTooLong(pos~ : Int)
    UnsupportedWireType(pos~ : Int, Int)
    LengthOutOfRange(pos~ : Int, Int)
    } derive(Eq,
    Debug
    )

    Malformed protobuf input. pos is the absolute byte offset in the buffer.

    PbfError::equal

    fn PbfError::equal(PbfError, PbfError) -> Bool

    PbfError::not_equal

    fn PbfError::not_equal(x : PbfError, y : PbfError) -> Bool

    PbfError::to_repr

    Reader

    pub struct Reader {
    // private fields
    }

    Zero-copy protobuf cursor. Reads never raise (in wasm-gc a call that may raise allocates): a failed read records the first error, returns 0 and jumps to the end so at_end loops stop. Check error once at a boundary.

    Reader::at_end

    fn Reader::at_end(self : Reader) -> Bool

    Whether nothing is left: the loop condition of a message, true after an error too.

    Reader::enter

    fn Reader::enter(self : Reader) -> Int

    Narrows the reader to the next length-delimited field, without allocating. Pass the returned token to exit once the field has been read.

    Reader::error

    fn Reader::error(self : Reader) -> PbfError?

    The first error since the last reset.

    Reader::exit

    fn Reader::exit(self : Reader, token : Int) -> Unit

    Leaves the field entered with enter, skipping whatever was not read.

    Reader::failed

    fn Reader::failed(self : Reader) -> Bool

    Whether a read has failed since the last reset.

    Reader::new

    fn Reader::new(view : BytesView) -> Reader

    A reader over view, at its start.

    Reader::pos

    fn Reader::pos(self : Reader) -> Int

    Absolute offset of the next byte to read.

    Reader::read_bool

    fn Reader::read_bool(self : Reader) -> Bool

    Reads a bool: any varint but 0 is true.

    Reader::read_bytes

    fn Reader::read_bytes(self : Reader) -> BytesView

    Reads a length-delimited field as a view of the original buffer (no copy).

    Reader::read_double

    fn Reader::read_double(self : Reader) -> Double

    Reads a double: IEEE 754 bits as a fixed64.

    Reader::read_fixed32

    fn Reader::read_fixed32(self : Reader) -> UInt

    Reads a little-endian fixed32.

    Reader::read_fixed64

    fn Reader::read_fixed64(self : Reader) -> UInt64

    Reads a little-endian fixed64.

    Reader::read_float

    fn Reader::read_float(self : Reader) -> Float

    Reads a float: IEEE 754 bits as a fixed32.

    Reader::read_int64

    fn Reader::read_int64(self : Reader) -> Int64

    Reads an int64: a varint in two's complement.

    Reader::read_len_start

    fn Reader::read_len_start(self : Reader) -> Int

    Skips over a length-delimited field and returns its absolute start offset; its end is pos() afterwards.

    Reader::read_sint64

    fn Reader::read_sint64(self : Reader) -> Int64

    Reads a zigzag-encoded sint64.

    Reader::read_string

    fn Reader::read_string(self : Reader) -> String

    Reads a UTF-8 string; invalid sequences become U+FFFD, like TextDecoder.

    Reader::read_svarint32

    fn Reader::read_svarint32(self : Reader) -> Int

    Reads a zigzag-encoded sint32.

    Reader::read_tag

    fn Reader::read_tag(self : Reader) -> Int

    Reads a field tag (field << 3 | wire); see tag and WIRE_*. Group wire types are an error.

    Reader::read_varint32

    fn Reader::read_varint32(self : Reader) -> Int

    Reads a varint and keeps its low 32 bits, as protobuf does for uint32.

    Reader::read_varint64

    fn Reader::read_varint64(self : Reader) -> UInt64

    Reads a varint of at most 10 bytes; longer is an error.

    Reader::remaining

    fn Reader::remaining(self : Reader) -> Int

    Bytes left in the current range.

    Reader::reset

    fn Reader::reset(self : Reader, start : Int, end : Int) -> Unit

    Repositions this reader over [start, end) of the same buffer and clears its error; offsets are absolute, as returned by pos. Reusing one reader avoids an allocation per message. Out-of-range bounds become an error.

    Reader::skip

    fn Reader::skip(self : Reader, tag : Int) -> Unit

    Skips the value of a field whose tag was just read.

    Writer

    pub struct Writer {
    // private fields
    }

    Minimal protobuf encoder, for building test data and fixtures.

    Writer::new

    fn Writer::new() -> Writer

    Writer::to_bytes

    fn Writer::to_bytes(self : Writer) -> Bytes

    Writer::write_bytes

    fn Writer::write_bytes(self : Writer, b : BytesView) -> Unit

    Writer::write_fixed32

    fn Writer::write_fixed32(self : Writer, v : UInt) -> Unit

    Writer::write_fixed64

    fn Writer::write_fixed64(self : Writer, v : UInt64) -> Unit

    Writer::write_sint64

    fn Writer::write_sint64(self : Writer, v : Int64) -> Unit

    Writer::write_string

    fn Writer::write_string(self : Writer, s : String) -> Unit

    Writer::write_svarint32

    fn Writer::write_svarint32(self : Writer, v : Int) -> Unit

    Writer::write_tag

    fn Writer::write_tag(self : Writer, field : Int, wire : Int) -> Unit

    Writer::write_varint

    fn Writer::write_varint(self : Writer, v : UInt64) -> Unit

    Writer::write_varint32

    fn Writer::write_varint32(self : Writer, v : Int) -> Unit

    Writes a 32-bit value as protobuf does: negatives take 10 bytes.

    WIRE_FIXED32

    let WIRE_FIXED32 : Int

    WIRE_FIXED64

    let WIRE_FIXED64 : Int

    WIRE_LEN

    let WIRE_LEN : Int

    WIRE_VARINT

    let WIRE_VARINT : Int

    tag

    fn tag(field : Int, wire : Int) -> Int

    Encodes a field tag, e.g. for matching: tag(3, WIRE_LEN).