README
moonproto
Protocol Buffers for MoonBit.
protobuf
protocol-buffers
proto
proto2
proto3
editions
serialization
binary
codegen
protoc
descriptor
reflection
protojson
moonbit
Author
Version
0.1.1
License
Apache-2.0
Repository
Last updated
9 hours ago
Downloads
430
#moonproto
Protocol Buffers for MoonBit: the wire format, the descriptor model, binary,
JSON and text encodings, dynamic messages, the well-known types, a .proto
parser, and a protoc plugin that generates MoonBit message code.
Home · Guide · API reference · MoonCakes · Specification
#Status
The runtime, the encodings, the parser, the generator, the plugin, the
well-known types and the conformance harness are implemented. Generated
messages are ordinary MoonBit structs with real fields, and the well-known
package is generated by this repository's own plugin from the schemas protobuf
36.2 ships.
A few numbers that describe the current shape of the library:
| Wire types | 5 |
| Scalar kinds | 15 |
| Syntax shapes generated and checked | 4 (proto2, proto3, editions 2023, editions 2024) |
| Official binary and JSON conformance cases passing | 2737 / 2737 |
| Official text format conformance cases passing | 440 / 440 |
| Declared conformance omissions | 0 |
#Packages
| Package | Carries |
|---|---|
| wire | tags, varints, fixed-width numbers, length-delimited values, groups, zig-zag, and the reader that reports where it stopped |
| reflect | Value, Message, Enum, ListField, MapField, the descriptor records, and the global registries |
| desc | building runtime descriptors from the descriptor shape, and converting back |
| proto | binary encode and decode, size, merge, clone, equal, reset, required checks, extensions on the wire |
| json | the JSON mapping: 64-bit integers as strings, enums by name, base64 bytes, maps as objects, extensions in brackets, the Timestamp, Duration, wrapper, FieldMask and Any forms, and arbitrary JSON through Struct, Value and ListValue |
| text | the text format: fields and separators, braces and angle brackets, groups, lists, every literal form, comments, expanded Any, extensions in brackets |
| delim | varint size-delimited messages over a byte source |
| dynamic | messages, enums and extensions built at run time from descriptors |
| known | the well-known types; a thin layer over known/google/protobuf, which the plugin generated |
| parser | the .proto language: proto2, proto3 and editions, messages, maps, oneofs, groups, enums, services, reserved and extension ranges, features and custom options |
| gen | the plugin protocol and the MoonBit code generator: a request in, generated files out |
| conformance | the conformance protocol: read one format, write another, report the error |
| cmd/protoc-gen-mbt | the protoc plugin binary |
| cmd/conformance-mbt | the conformance testee: 4-byte length frames on standard input and output |
| cmd/interop-mbt | the MoonBit half of the bidirectional interop program |
The root package is empty; a program imports the subpackages it uses.
#Generated messages
A generated message is a struct whose fields are the protobuf fields. There
is no hidden dynamic message behind it:
pub struct Envelope {
mut i32 : Int32 // proto3, implicit presence
mut name : String? // explicit presence
nums : Array[Int32] // repeated
counts : Map[String, Int32] // map
mut pick : Envelope_Pick // oneof, as an enum
mut unknown_fields : Bytes
}The struct implements reflect.Message, so encode, decode, size, merge, JSON
and text all reach its fields through one trait. Typed views wrap Array and
Map as the reflective ListField and MapField without copying. A companion
reflect.MessageType registers the type by name, which is how Any resolves
its payload. Unknown fields are kept on the struct and re-emitted on the next
encode.
protoc-gen-mbt advertises editions support and resolves the unresolved
FeatureSet values protoc sends from the innermost scope outward. That is why
field_presence, repeated_field_encoding, message_encoding and enum_type
change the generated struct shape, the packing on the wire and enum openness.
#How it fits together
Descriptors are plain records. desc builds them from the descriptor shape;
the encoders walk the descriptor and never look at a struct's layout.
Cross-type references are fully-qualified names resolved through a symbol table
when the file is built. A schema can therefore be read, filled and encoded with
no generated code at all:
let schema =
"syntax = \"proto3\";\n" +
"package hello;\n" +
"message Request { string name = 1; int32 count = 2; repeated int32 nums = 3; }\n"
let parsed = @parser.parse("hello.proto", schema) catch { _ => abort("bad schema") }
let file = @desc.build_file(parsed) catch { _ => abort("bad descriptor") }
let request = file.messages[0]
assert_eq(request.full_name, "hello.Request")
assert_eq(request.fields.length(), 3)A dynamic message holds the values, and every encoder takes the same
reflect.Message:
let message = @dynamic.Message::new(request)
let name = request.field_by_name("name").unwrap()
message.set(name, @reflect.String("moon"))
let bytes = @proto.encode(message) catch { _ => abort("encode failed") }
assert_eq(bytes.length(), 6)
let back = @dynamic.Message::new(request)
@proto.decode(bytes, back) catch { _ => abort("decode failed") }
assert_eq(back.get(name).string(), "moon")The JSON mapping is the same walk with the mapping rules applied — 64-bit
integers as strings, enums by name, bytes in base64:
let json = @json.encode(message) catch { _ => abort("JSON failed") }
assert_eq(json, "{\"name\":\"moon\"}")The text format carries the same values with its own grammar:
let text = @text.encode(message) catch { _ => abort("text failed") }
assert_eq(text, "name: \"moon\"\n")A generated message takes the same path. A well-known type is a generated
message like any other, and it brings its own JSON form:
@known.register_all()
let stamp = @protobuf.Timestamp::new()
stamp.set_seconds(1_700_000_000L)
let stamp_json = @json.encode(stamp) catch { _ => abort("JSON failed") }
assert_eq(stamp_json, "\"2023-11-14T22:13:20Z\"")#Parsing and generating
parser.parse(file_name, source) returns the descriptor shape, which desc
turns into runtime descriptors. gen.generate_source(file) returns MoonBit
source for one file, and gen.plugin_main speaks the protoc protocol: a
serialized CodeGeneratorRequest in, a serialized CodeGeneratorResponse out.
cmd/protoc-gen-mbt wires that to standard input and output.
To generate into a project:
protoc -I . --plugin=protoc-gen-mbt=path/to/protoc-gen-mbt \
--mbt_out=src hello.proto#Editions and options
edition = "2023" and edition = "2024" are parsed, and option features.*
(or the same keys in a field's brackets) are resolved from the innermost scope
outward. The four that change what the runtime does are applied:
field_presence sets presence, repeated_field_encoding sets packing,
message_encoding = DELIMITED makes a field a group on the wire, and
enum_type = CLOSED closes an enum. Custom options written as
option (my.opt) = value are kept and reachable through
FileDescriptor::option_of(declaration, name).
#Consistency
The official protobuf 36.2 conformance_test_runner drives
cmd/conformance-mbt over 4-byte little-endian length frames. Both checked-in
failure lists live in tests/conformance/.
| Suite | Cases | Passed | Skipped | Expected failures | Unexpected failures |
|---|---|---|---|---|---|
| Binary and JSON (failure_list.txt) | 2737 | 2737 | 0 | 0 | 0 |
| Text format (text_format_failure_list.txt) | 440 | 440 | 0 | 0 | 0 |
Both failure lists are empty: no case in either suite is waived. The text
format suite is the one that exercises the grammar rather than the message
model, and it covers:
| Area | What is read |
|---|---|
| Fields | name: value, name value for message values, ; or , closing a field, { ... } and < ... > bodies |
| Groups | the group's type name or its own field name, in any case, with or without a colon, as a declared field or as an extension |
| Repeated | one element per name, or a bracketed list with , between elements |
| Scalars | decimal, 0x hexadecimal and leading-zero octal integers with a range check per width; decimal and exponent floats with an optional f suffix, .5 and -.5 forms, case-insensitive inf/infinity/nan, overflow to infinity and underflow to a signed zero |
| Strings | single or double quotes, adjacent literals joined, \a \b \f \n \r \t \v \? \\ \' \", \xNN, octal and \u/\U escapes, no raw line feed |
| Names | extensions and expanded Any payloads in brackets, where whitespace and comments are dropped and a % must introduce two hex digits |
| Refusals | an out-of-range integer, a hexadecimal or octal float literal, a number a closed enum does not declare, and a name the schema neither declares nor reserves |
#Bidirectional interop
tests/interop/interop.proto is shared by three programs: the protobuf
compiler, a Go program built from protoc-gen-go v1.36.12 output, and
cmd/interop-mbt. Every program encodes the same fixture and decodes the
other's bytes, then re-encodes and compares. All 103 bytes agree, and each side
asserts every field value after decoding, including the repeated scalars, the
map, the nested message, the enum and the active oneof member.
#Generated code
scripts/check-codegen.sh generates and compiles four schemas with
moon check --deny-warn:
| Schema | Shape covered | Warnings |
|---|---|---|
| tests/shapes/proto2.proto | required, defaults, packed repeats, extension range and a top-level extension | 0 |
| tests/shapes/proto3.proto | every scalar, repeated messages, map, enum, oneof | 0 |
| tests/shapes/edition2023.proto | file-level implicit presence, per-field packed and expanded encoding | 0 |
| tests/shapes/edition2024.proto | current edition naming rules, explicit presence, a closed enum and a delimited (group-encoded) field | 0 |
#Mutation evidence
Three invariants are guarded by the test suite. Breaking each one turns the
following tests red:
| Invariant broken | Tests that fail |
|---|---|
| Varint ten-octet limit | protowire: the largest varint takes ten octets, an eleventh overflows; varint refuses an eleventh octet |
| Unknown-field preservation | proto: Marshal and Unmarshal keep a scalar alongside an unknown field; unknown fields survive a round-trip |
| 64-bit integers as JSON strings | protojson: 64-bit integers are strings on the wire; binary to JSON keeps every scalar and the composite fields; the scalar mapping and its fixed text |
Run scripts/mutation-check.sh to reproduce all three; it restores every
source file before it exits.
#Reproducing the results
| Script | Checks |
|---|---|
| scripts/setup-protoc.sh | pins protoc 36.2 and prints PROTOC / PROTOC_INCLUDE |
| scripts/conformance.sh | runs the official runner with both failure lists, or the framing self-check when no runner is present |
| scripts/check-failure-lists.sh | the binary/JSON list is empty; every text-format entry has a name and a reason |
| scripts/check-conformance-types.sh | the checked-in descriptors match the pinned descriptor set |
| scripts/vendor_conformance_descriptors.sh | regenerates that descriptor set from a protobuf 36.2 checkout |
| scripts/check-codegen.sh | the four generated schemas compile warning-free |
| scripts/check-gointerop.sh | the checked-in Go schema is exactly what protoc-gen-go v1.36.12 emits |
| scripts/interop.sh | the fixture's reference bytes are what protoc itself encodes |
| scripts/interop-go.sh | Go and MoonBit encode and decode each other's bytes |
| scripts/mutation-check.sh | the three mutation checks above |
| scripts/generate-known.sh | regenerates the well-known package from the protobuf includes |
| scripts/check_snippets.py | every MoonBit block in this file and in the skill compiles against the source |
#Design notes
- docs/design/inventory.md — every capability, one row each, with where it lives.
- docs/design/interface.md — the types and signatures the implementation was written against, and the mechanism that replaces runtime field access.
- docs/skills/moonproto/SKILL.md — the short reference for an agent using this library.
#Licence
Apache-2.0.
Install
Download zipProtocol Buffers for MoonBit.
protobufprotocol-buffersprotoproto2proto3editionsserializationbinarycodegenprotocdescriptorreflectionprotojsonmoonbit
Apache-2.0
Updated 9 hours ago
430 downloads