The Wax language in MoonBit: parser, formatter, type checker, and wasm/wat emitters

wax

webassembly

wasm

compiler

parser

formatter

wat

moon add marianoguerra/wax@0.2.0

Download zip

Author

marianoguerra

Version

0.2.0

License

Apache-2.0

Repository

github.com/marianoguerra/wax-mb

Last updated

2 days ago

Downloads

README

#marianoguerra/wax

The Wax language in MoonBit: a parser, a formatter, a type checker, and emitters for both the binary and the text form of WebAssembly.

Wax is a Rust-like surface syntax for WebAssembly. It reads as a programming language:

#[export]
fn add(x: i32, y: i32) -> i32 {
    x + y;
}

and compiles to the same bytecode as the equivalent stack-machine WAT.

This is a port of the reference implementation, written against it as a specification rather than as an inspiration: a differential harness runs both over a ~2000-module corpus and gates on reprint parity, wasm equivalence and diagnostic parity. See the repository for what that suite covers and where the port has not reached parity yet.

No dependencies. This module imports nothing outside moonbitlang/core. The command-line tool, which needs filesystem access, is a separate module: marianoguerra/wax-cli.

Every example below is a test in facade_test.mbt and ast/build/build_test.mbt, so moon test is what says this page is still true.

#Source in, wasm out

let src =
  #|#[export]
  #|fn add(x: i32, y: i32) -> i32 { x + y; }
  #|
let bytes = match @wax.compile_string(src) {
  Ok(b) => b
  Err(diagnostics) => ...   // nothing compiled; these say why
}

Err carries the diagnostics that stopped it and nothing else: a compile that reports an error produces no bytes at all, rather than bytes derived from a module known to be wrong.

A rejected module comes back as Err; a module that checked and then would not lower raises CompileError, which is a bug in this port rather than in the input. It names its three stages — Lower, Encode, Wat — so a caller can match on which one, and Show renders each with its own span.

compile_string_to_wat takes the same route as far as the lowered module and prints it instead of encoding it, so the two forms cannot describe different modules:

let text = @wax.compile_string_to_wat("fn one() -> i32 { 1; }")
// (func $one (result i32) (i32.const 1))

#Source in, source out

The formatter is a front-end operation: it does not type-check, because a module that does not type-check still has a canonical layout. Err therefore only ever holds syntax errors.

@wax.format_string("fn f()->i32{1;}")
// Ok("fn f() -> i32 {\n    1;\n}")

#AST in, wasm out

This is the path a code generator wants, and the reason the module is laid out the way it is. Import marianoguerra/wax/compile rather than marianoguerra/wax, and the lexer, the token table and the generated LR parser are never compiled — about 15k lines, most of it a table nobody wrote by hand.

marianoguerra/wax/ast/build has smart constructors for the module fields; anything it does not cover is a plain @ast value written out by hand, and the two mix freely.

let i32_ : @wasm_types.ValType[@ast.Ident] = I32
let m : @ast.LocModule = [
  @build.func(
    "add",
    params=[(Some("x"), i32_), (Some("y"), i32_)],
    results=[i32_],
    body=[
      @ast.no_loc_instr(
        BinOpI(
          @basic.no_loc(Add),
          @ast.no_loc_instr(Get(@build.ident("x"))),
          @ast.no_loc_instr(Get(@build.ident("y"))),
        ),
      ),
    ],
    attributes=[@build.exported()],
  ),
]

let session = @compile.Session::new()
let checked = session.check(m)
if @compile.rejected(session.reports()) {
  // report and stop
} else {
  let bytes = checked.to_bytes()   // or checked.to_wat()
}

check returns a Checked even for a module it rejected, because whether a run is rejected is not its decision: a named warning is hidden, shown or promoted to an error by a @warning.Policy the caller owns. Ask reports(policy?) for the diagnostics under that policy, rejected whether they stop you, and only then lower.

Give your builders locations. Every constructor in ast/build takes at~ : @basic.Location, defaulting to @basic.dummy_loc. Mapping your own source spans into it costs one argument and buys the entire diagnostic renderer — the snippet, the caret, the related labels, the quick fixes — pointing at your syntax rather than at nothing.

A binding's location has to be distinct. A local's wasm slot is keyed by the offset its name was written at — that is what lets a shadowing let take a new slot while the old name is still readable in its own initializer. So the names in two lets, or in two match arms, of one function must not share a span. Nothing parsed can violate that; a generated tree can, by building every identifier at dummy_loc. Doing so is refused rather than emitted (@wasm.AmbiguousBinding), since the module it would produce is wrong rather than invalid. A generator with no source text of its own mints spans instead:

let s = @build.Spans::new()          // or Spans::new(fname="my.dsl")
let body = [
  s.instr(Let([(Some(s.ident("a")), Some(i32_))], Some(s.instr(Int("1"))))),
  s.instr(Get(s.ident("a"))),
]

Each span is a line of its own, so a diagnostic names the node (File"<generated>", line 7) even with no text behind it.

#The packages

package
wax	the facade above: parse, format, compile, one call each
wax/compile	Session and Checked: the AST-first pipeline, no front end
wax/ast, wax/ast/build	the syntax tree, and constructors for it
wax/syntax/{lexer,tokens,parser,trivia}	the front end
wax/fmt	the formatter
wax/check, wax/check/{env,store,infer,members}	the type checker
wax/emit/wasm	Wax → the wasm module model
wax/wasm/{types,bin,wat,simd,atomics}	the wasm model, its encoder and its printer — no Wax anywhere in them
wax/{basic,diagnostic,message,printer,colors,warning,feature,cond,unicode}	spans, diagnostics and layout

wax/internal/* is private to this module: MoonBit refuses the import from outside it.

#Stability

Version 0.2. Treated as stable, and changed only with a version bump: the wax facade, wax/compile, wax/ast and wax/ast/build, wax/wasm/*, wax/basic, wax/diagnostic, wax/message, wax/warning, wax/feature, and the entry points @grammar.parse_string / parse_recover, @output.render, @typing.check_module and @to_wasm.lower_module.

Everything else is 0.x-mutable.

Before the first release the surface was cut from 1117 public names to 685 — wax/check alone from 349 to 103 — on one rule: a name nothing outside its own package referenced was not a promise, whatever it was doing in a .mbti. What is left in the checker is mostly reached by its own tests. tools/api_audit.py in the repository reports the remainder, and that surface is being narrowed rather than widened.

#Licence

Apache-2.0, matching the reference implementation this is ported from. See NOTICE for the vendored wasm_core encoder.

#
Checked

source

using @marianoguerra/wax/compile { type Checked }

The AST-first pipeline, re-exported from marianoguerra/wax/compile.

These names are the reason that package exists separately: a code generator imports it directly and never pays for the parser.

#
CompileError

source

using @marianoguerra/wax/compile { type CompileError }

The AST-first pipeline, re-exported from marianoguerra/wax/compile.

These names are the reason that package exists separately: a code generator imports it directly and never pays for the parser.

#
ParseResult

source

using @marianoguerra/wax/syntax/parser { type ParseResult }

The front end, re-exported.

parse_string stops at the first syntax error; parse_recover reports every error it can resynchronize past, at the price of a tree that is partly a guess about what the author meant. Both return a ParseResult whose module_ is None only when nothing could be recovered.

#
Session

source

using @marianoguerra/wax/compile { type Session }

The AST-first pipeline, re-exported from marianoguerra/wax/compile.

These names are the reason that package exists separately: a code generator imports it directly and never pays for the parser.

#
apply_policy

source

fn apply_policy(policy : @marianoguerra/wax/warning.Policy
Policy, collected : Array[@marianoguerra/wax/diagnostic.Diagnostic
Diagnostic]) -> Array[@marianoguerra/wax/diagnostic.Diagnostic
Diagnostic]

Resolve every named warning in collected against policy.

The collector buffers diagnostics RAW: the policy belongs to whoever finally re-reports them. Order is preserved, hidden warnings are dropped, and a warning promoted to an error comes back with its severity replaced rather than duplicated.

#
compile_string

source

fn compile_string(src : String, fname? : String, features? : @marianoguerra/wax/feature.Set
Set, policy? : @marianoguerra/wax/warning.Policy
Policy) -> Result[Bytes, Array[@marianoguerra/wax/diagnostic.Diagnostic
Diagnostic]] raise @marianoguerra/wax/compile.CompileError
CompileError

Parse, check and compile Wax source to a .wasm binary.

Err carries the diagnostics that stopped it -- syntax errors, or whatever the type checker rejected under policy. Nothing is returned alongside them: a compile that reports an error produces no bytes at all, rather than bytes derived from a module known to be wrong.

Warnings never reach the caller here, because a compile reports what stops it producing a module and a style warning does not. Use Session directly, or the check command's warn_unused, to see those.

#
compile_string_to_wat

source

fn compile_string_to_wat(src : String, fname? : String, features? : @marianoguerra/wax/feature.Set
Set, policy? : @marianoguerra/wax/warning.Policy
Policy) -> Result[String, Array[@marianoguerra/wax/diagnostic.Diagnostic
Diagnostic]] raise @marianoguerra/wax/compile.CompileError
CompileError

Parse, check and print Wax source in the WebAssembly text format.

The same lowering compile_string encodes, printed instead of encoded, so the two forms describe the same module by construction. Comments survive: they come from the parse's trivia table and are re-delimited as WAT comments.

#
format_string

source

fn format_string(src : String, fname? : String, theme? : @marianoguerra/wax/colors.Theme
Theme) -> Result[String, Array[@marianoguerra/wax/diagnostic.Diagnostic
Diagnostic]]

Reformat Wax source.

The type checker does not run: reformatting is a front-end operation, and a module that does not type-check still has a canonical layout. Err therefore only ever holds syntax errors.

#
parse_recover

source

fn parse_recover(src : String, fname? : String) -> @marianoguerra/wax/syntax/parser.ParseResult
ParseResult

A parse that reports every syntax error it can resynchronize past.

The module is best-effort: whatever the actions built from the parts that did parse, or None if recovery never got the automaton back to an accept.

#
parse_string

source

fn parse_string(src : String, fname? : String) -> @marianoguerra/wax/syntax/parser.ParseResult
ParseResult

Parse src.

Three failure paths, all funnelled into errors:

a LEXICAL error, which stops the scan (lexer.tokens_from_string);
a SEMANTIC error raised from a grammar action -- recorded rather than raised, since MoonYacc's actions cannot propagate one;
a SYNTAX error from the automaton itself, ParseError.

#
rejected

source

fn rejected(diagnostics : Array[@marianoguerra/wax/diagnostic.Diagnostic
Diagnostic]) -> Bool

Whether any of these diagnostics rejects the module.

Apply the policy first: a warning promoted to an error rejects, and one hidden by the policy does not.

#
render

source

fn render(m : Array[@marianoguerra/wax/basic.Annotated
Annotated[@marianoguerra/wax/ast.ModuleField
ModuleField[@marianoguerra/wax/basic.Location
Location], @marianoguerra/wax/basic.Location
Location]], trivia_ctx : @marianoguerra/wax/syntax/trivia.Context
Context, theme? : @marianoguerra/wax/colors.Theme
Theme) -> String

Format a parsed module as Wax source.

TWO PASSES, and they are not optional. The first runs the printer with output discarded, purely to record which spans it looks up; associate then restricts comment attachment to exactly those. Attaching first and printing second would let a comment land on a node the printer skips, and vanish. The dry pass takes no theme: it emits nothing, so colouring it would only cost work. Width is irrelevant there too -- the traversal looks up the same spans however the lines break.