🥮 mooncakes.io

    moonproto

    Protocol Buffers for MoonBit.

    protobuf
    protocol-buffers
    proto
    proto2
    proto3
    editions
    serialization
    binary
    codegen
    protoc
    descriptor
    reflection
    protojson
    moonbit
    Download zip
    Version
    0.1.2
    License
    Apache-2.0
    Repository
    Last updated
    2 days ago
    Downloads
    1K

    #moonproto

    Protocol Buffers for MoonBit: the wire format, the descriptor model, binary, JSON and text encodings, dynamic messages, the well-known types, a .proto parser, and a protoc plugin that generates MoonBit message code.

    Home · Guide · API reference · MoonCakes · Specification

    #Status

    The runtime, the encodings, the parser, the generator, the plugin, the well-known types and the conformance harness are implemented. Generated messages are ordinary MoonBit structs with real fields, and the well-known package is generated by this repository's own plugin from the schemas protobuf 36.2 ships.

    A few numbers that describe the current shape of the library:

    Wire types5
    Scalar kinds15
    Syntax shapes generated and checked4 (proto2, proto3, editions 2023, editions 2024)
    Official binary and JSON conformance cases passing2755 / 2755
    Official text format conformance cases passing440 / 440
    Declared conformance omissions0

    #Packages

    PackageCarries
    wiretags, varints, fixed-width numbers, length-delimited values, groups, zig-zag, and the reader that reports where it stopped
    reflectValue, Message, Enum, ListField, MapField, the descriptor records, and the global registries
    descbuilding runtime descriptors from the descriptor shape, and converting back
    protobinary encode and decode, size, merge, clone, equal, reset, required checks, extensions on the wire
    jsonthe JSON mapping: 64-bit integers as strings, enums by name, base64 bytes, maps as objects, extensions in brackets, the Timestamp, Duration, wrapper, FieldMask and Any forms, and arbitrary JSON through Struct, Value and ListValue
    textthe text format: fields and separators, braces and angle brackets, groups, lists, every literal form, comments, expanded Any, extensions in brackets
    delimvarint size-delimited messages over a byte source
    dynamicmessages, enums and extensions built at run time from descriptors
    knownthe well-known types; a thin layer over known/google/protobuf, which the plugin generated
    parserthe .proto language: proto2, proto3 and editions, messages, maps, oneofs, groups, enums, services, reserved and extension ranges, features and custom options
    genthe plugin protocol and the MoonBit code generator: a request in, generated files out
    conformancethe conformance protocol: read one format, write another, report the error
    cmd/protoc-gen-mbtthe protoc plugin binary
    cmd/conformance-mbtthe conformance testee: 4-byte length frames on standard input and output
    cmd/interop-mbtthe MoonBit half of the bidirectional interop program

    The root package is empty; a program imports the subpackages it uses.

    #Generated messages

    A generated message is a struct whose fields are the protobuf fields. There is no hidden dynamic message behind it:

    pub struct Envelope {
      mut i32 : Int32              // proto3, implicit presence
      mut name : String?           // explicit presence
      nums : Array[Int32]          // repeated
      counts : Map[String, Int32]  // map
      mut pick : Envelope_Pick     // oneof, as an enum
      mut unknown_fields : Bytes
    }

    The struct implements reflect.Message, so encode, decode, size, merge, JSON and text all reach its fields through one trait. Typed views wrap Array and Map as the reflective ListField and MapField without copying. A companion reflect.MessageType registers the type by name, which is how Any resolves its payload. Unknown fields are kept on the struct and re-emitted on the next encode.

    protoc-gen-mbt advertises editions support and resolves the unresolved FeatureSet values protoc sends from the innermost scope outward. That is why field_presence, repeated_field_encoding, message_encoding and enum_type change the generated struct shape, the packing on the wire and enum openness.

    #Presence

    A field with presence has a set state of its own; a field without presence does not, and the wire has no way to record one. The library states the rule once, in reflect.is_populated, and every encoder asks it: a scalar without presence is written only when its value differs from the zero of its kind, so writing a zero and never mentioning the field produce the same bytes and the same JSON.

    let probe_schema =
      "syntax = \"proto3\";\n" +
      "package p;\n" +
      "message Probe { int32 i = 1; optional int32 o = 2; }\n"
    let shape = @parser.parse("probe.proto", probe_schema) catch {
      _ => abort("bad schema")
    }
    let built = @desc.build_file(shape) catch { _ => abort("bad descriptor") }
    let descriptor = built.messages[0]
    let probe = @dynamic.Message::new(descriptor)
    let i = descriptor.field_by_name("i").unwrap()
    let o = descriptor.field_by_name("o").unwrap()
    probe.set(i, @reflect.Int32(0))
    assert_eq(probe.has(i), false)
    assert_eq(@proto.encode(probe) catch { _ => abort("encode failed") }, b"")
    assert_eq(@json.encode(probe) catch { _ => abort("JSON failed") }, "{}")
    probe.set(o, @reflect.Int32(0))
    assert_eq(probe.has(o), true)
    assert_eq(@proto.encode(probe) catch { _ => abort("encode failed") }, b"\x10\x00")
    assert_eq(@json.encode(probe) catch { _ => abort("JSON failed") }, "{\"o\":0}")

    Negative zero is a value rather than a zero, because its sign bit is part of it: a float or double field holding -0.0 is written in both formats and reads back as one. A field that arrives carrying an explicit zero is read, reported as not set, and dropped by the next write.

    #How it fits together

    Descriptors are plain records. desc builds them from the descriptor shape; the encoders walk the descriptor and never look at a struct's layout. Cross-type references are fully-qualified names resolved through a symbol table when the file is built. A schema can therefore be read, filled and encoded with no generated code at all:

    let schema =
      "syntax = \"proto3\";\n" +
      "package hello;\n" +
      "message Request { string name = 1; int32 count = 2; repeated int32 nums = 3; }\n"
    let parsed = @parser.parse("hello.proto", schema) catch { _ => abort("bad schema") }
    let file = @desc.build_file(parsed) catch { _ => abort("bad descriptor") }
    let request = file.messages[0]
    assert_eq(request.full_name, "hello.Request")
    assert_eq(request.fields.length(), 3)

    A dynamic message holds the values, and every encoder takes the same reflect.Message:

    let message = @dynamic.Message::new(request)
    let name = request.field_by_name("name").unwrap()
    message.set(name, @reflect.String("moon"))
    let bytes = @proto.encode(message) catch { _ => abort("encode failed") }
    assert_eq(bytes.length(), 6)

    let back = @dynamic.Message::new(request)
    @proto.decode(bytes, back) catch { _ => abort("decode failed") }
    assert_eq(back.get(name).string(), "moon")

    The JSON mapping is the same walk with the mapping rules applied — 64-bit integers as strings, enums by name, bytes in base64:

    let json = @json.encode(message) catch { _ => abort("JSON failed") }
    assert_eq(json, "{\"name\":\"moon\"}")

    The text format carries the same values with its own grammar:

    let text = @text.encode(message) catch { _ => abort("text failed") }
    assert_eq(text, "name: \"moon\"\n")

    A generated message takes the same path. A well-known type is a generated message like any other, and it brings its own JSON form:

    @known.register_all()
    let stamp = @protobuf.Timestamp::new()
    stamp.set_seconds(1_700_000_000L)
    let stamp_json = @json.encode(stamp) catch { _ => abort("JSON failed") }
    assert_eq(stamp_json, "\"2023-11-14T22:13:20Z\"")

    #Parsing and generating

    parser.parse(file_name, source) returns the descriptor shape, which desc turns into runtime descriptors. gen.generate_source(file) returns MoonBit source for one file, and gen.plugin_main speaks the protoc protocol: a serialized CodeGeneratorRequest in, a serialized CodeGeneratorResponse out. cmd/protoc-gen-mbt wires that to standard input and output.

    To generate into a project:

    protoc -I . --plugin=protoc-gen-mbt=path/to/protoc-gen-mbt \
  --mbt_out=src hello.proto

    #Editions and options

    edition = "2023" and edition = "2024" are parsed, and option features.* (or the same keys in a field's brackets) are resolved from the innermost scope outward. The four that change what the runtime does are applied: field_presence sets presence, repeated_field_encoding sets packing, message_encoding = DELIMITED makes a field a group on the wire, and enum_type = CLOSED closes an enum. Custom options written as option (my.opt) = value are kept and reachable through FileDescriptor::option_of(declaration, name).

    #Consistency

    The official protobuf 36.2 conformance_test_runner drives cmd/conformance-mbt over 4-byte little-endian length frames. Both checked-in failure lists live in tests/conformance/.

    SuiteCasesPassedSkippedExpected failuresUnexpected failures
    Binary and JSON (failure_list.txt)27552755000
    Text format (text_format_failure_list.txt)440440000

    Both failure lists are empty: no case in either suite is waived. The text format suite is the one that exercises the grammar rather than the message model, and it covers:

    AreaWhat is read
    Fieldsname: value, name value for message values, ; or , closing a field, { ... } and < ... > bodies
    Groupsthe group's type name or its own field name, in any case, with or without a colon, as a declared field or as an extension
    Repeatedone element per name, or a bracketed list with , between elements
    Scalarsdecimal, 0x hexadecimal and leading-zero octal integers with a range check per width; decimal and exponent floats with an optional f suffix, .5 and -.5 forms, case-insensitive inf/infinity/nan, overflow to infinity and underflow to a signed zero
    Stringssingle or double quotes, adjacent literals joined, \a \b \f \n \r \t \v \? \\ \' \", \xNN, octal and \u/\U escapes, no raw line feed
    Namesextensions and expanded Any payloads in brackets, where whitespace and comments are dropped and a % must introduce two hex digits
    Refusalsan out-of-range integer, a hexadecimal or octal float literal, a number a closed enum does not declare, and a name the schema neither declares nor reserves

    #Bidirectional interop

    tests/interop/interop.proto is shared by three programs: the protobuf compiler, a Go program built from protoc-gen-go v1.36.12 output, and cmd/interop-mbt. Every program encodes the same fixture and decodes the other's bytes, then re-encodes and compares. All 103 bytes agree, and each side asserts every field value after decoding, including the repeated scalars, the map, the nested message, the enum and the active oneof member.

    #Generated code

    scripts/check-codegen.sh generates and compiles four schemas with moon check --deny-warn:

    SchemaShape coveredWarnings
    tests/shapes/proto2.protorequired, defaults, packed repeats, extension range and a top-level extension0
    tests/shapes/proto3.protoevery scalar, repeated messages, map, enum, oneof0
    tests/shapes/edition2023.protofile-level implicit presence, per-field packed and expanded encoding0
    tests/shapes/edition2024.protocurrent edition naming rules, explicit presence, a closed enum and a delimited (group-encoded) field0

    #Mutation evidence

    Three invariants are guarded by the test suite. Breaking each one turns the following tests red:

    Invariant brokenTests that fail
    Varint ten-octet limitprotowire: the largest varint takes ten octets, an eleventh overflows; varint refuses an eleventh octet
    Unknown-field preservationproto: Marshal and Unmarshal keep a scalar alongside an unknown field; unknown fields survive a round-trip
    64-bit integers as JSON stringsprotojson: 64-bit integers are strings on the wire; binary to JSON keeps every scalar and the composite fields; the scalar mapping and its fixed text
    A zero without presence stays off the wirebinary: a zero without presence writes no bytes; binary: a zero read from the wire does not come back; binary and JSON leave the same zeros out
    Negative zero keeps its signbinary: negative zero keeps its sign on the wire; a negative zero is written by both encodings

    Run scripts/mutation-check.sh to reproduce all five; it restores every source file before it exits.

    #Reproducing the results

    ScriptChecks
    scripts/setup-protoc.shpins protoc 36.2 and prints PROTOC / PROTOC_INCLUDE
    scripts/conformance.shruns the official runner with both failure lists, or the framing self-check when no runner is present
    scripts/check-failure-lists.shthe binary/JSON list is empty; every text-format entry has a name and a reason
    scripts/check-conformance-types.shthe checked-in descriptors match the pinned descriptor set
    scripts/vendor_conformance_descriptors.shregenerates that descriptor set from a protobuf 36.2 checkout
    scripts/check-codegen.shthe four generated schemas compile warning-free
    scripts/check-gointerop.shthe checked-in Go schema is exactly what protoc-gen-go v1.36.12 emits
    scripts/interop.shthe fixture's reference bytes are what protoc itself encodes
    scripts/interop-go.shGo and MoonBit encode and decode each other's bytes
    scripts/mutation-check.shthe three mutation checks above
    scripts/generate-known.shregenerates the well-known package from the protobuf includes
    scripts/check_snippets.pyevery MoonBit block in this file and in the skill compiles against the source

    #Agent skill

    docs/skills/moonproto/SKILL.md is the short reference for an agent using this library: what to import, the two or three things it will be asked to do, and the rules that are easy to get wrong. It ships with every release.

    #Licence

    Apache-2.0.