nanocake

Composable, type-safe AST and compiler-pass generation for MoonBit.

compiler
ast
nanopass
code-generation
sexp
moon add YumeXi/nanocake@0.3.0
Download zip
Author
Version
0.3.0
License
BSD-2-Clause
Last updated
5 hours ago
Downloads
4
README

#YumeXi/nanocake

A metaprogramming framework for defining composable, type-safe AST transformations.

Given MoonBit enums annotated with #nanopass.language, the library generates unified Tree types that merge shared constructors across language extensions, plus S-expression parsers/unparsers and transformation-pass scaffolding for the resulting ASTs.

#Architecture

┌─────────────┐ #nanopass.language │ meta_parser │ parse annotations, resolve extends chains, └──────┬──────┘ apply #nano.remove, resolve entries/.mbti │ NanoLangDef[] ▼ ┌─────────────┐ │ language │ expand inheritance -> a layout, then └──────┬──────┘ generate the unified Tree + wrappers │ GeneratedLanguageLayout ┌───────────┼───────────┐ ▼ ▼ ┌─────────────┐ ┌─────────────┐ │unparser/gen │ │ pass │ transform/fold scaffolding, └──────┬──────┘ └─────────────┘ smart constructors, PassM │ from_sexp / to_sexp_doc ▼ ┌─────────────┐ │unparser/sexp│ S-expression runtime: parse, doc render, form matching └─────────────┘

  • meta_parser — Parses #nanopass.language / #nanopass.nonterminal and #nano.* annotations from source files. Resolves language definitions, extends chains, #nano.remove deletions, effective entries, and .mbti interfaces.
  • language — Expands each language against its parent (inheritance, override, and #nano.remove) into a GeneratedLanguageLayout, then generates the AST scaffolding: unified syntax trees, per-language wrappers, extension enums, and terminal aliases.
  • unparser/sexp — S-expression runtime shared by generated code: a parser, a pretty-printing document model, #nano.form pattern matching, and layered parse/decode/hook errors.
  • unparser/gen — Generates concrete from_sexp / to_sexp_doc / parse_sexp / unparse_sexp codecs from a language's layout, honoring custom surface syntax and hooks.
  • pass — Transformation-pass scaffolding built on the layout (smart constructors, catamorphisms, delayed-subtree rewrites, and the effectful PassM pipeline). See pass/README.mbt.md.

The poc package is kept as a fixture/example package for .mbti-backed tests. It is not part of the first-round public package split.

#Quick Start

#1. Define languages

///|
pub type LambdaVar = String

///|
#nanopasslanguage(name="Lambda")
pub(all) enum Expr {
Int(Int)
LambdaVar(LambdaVar)
Lam(LambdaVar, Expr)
App(Expr, Expr)
}

///|
#nanopasslanguage(name="SimplyTypedLambda", extends="Lambda")
enum TypedExpr {
True
False
Int(Int)
LambdaVar(LambdaVar)
If(cond~ : TypedExpr, TypedExpr, TypedExpr)
Lam(LambdaVar, Type, TypedExpr)
App(TypedExpr, TypedExpr)
}

Surface syntax can be declared alongside constructors with #nano.* attributes:

#nanoform("(lambda ($0 : $1) $2)")
#nanoform_only
Lam(LambdaVar, Type, TypedExpr)

#nanoinline // renders as ($0 $1) — no constructor head
App(TypedExpr, TypedExpr)

To remove an inherited constructor in a derived language, declare it at the top of the enum:

#nanopasslanguage(name="NoApp", extends="Lambda")
#nanoremove("App")
enum NoAppExpr {
// only Lam, Int, LambdaVar are inherited; App is dropped
}

#2. Generate AST scaffolding

let lang = @language.Language::from_file(
name="Lambda", path="poc/lang_def.mbt", mod="YumeXi/nanocake",
)
let impls = lang.gen()
@fmt.impls_to_string(impls) // format and write to file

This produces a unified Tree[Self_, LamE, TreeExt] enum, per-language structs (Expr, TypedExpr), extension enums (TypedExprExt), external terminal aliases, and stub parse/unparse entry functions.

#3. Generate S-expression codecs

let lang = @language.Language::from_file(...)
// Generate from_sexp / parse_sexp / to_sexp_doc / unparse_sexp per type.
let codec_impls = @gen.gen(lang)

The generator honours #nano.form, #nano.inline, #nano.infix, #nano.name surface syntax and #nano.parse_by / #nano.unparse_by hook escape hatches. It also emits a from_sexp_without_hooks variant so hooks can call back into the generated parser without infinite recursion.

#4. Write passes

See the pass package for transform/fold scaffolding and the PassM effectful pipeline combinator.

#Package Reference

#@meta_parser — Parse annotations

APIDescription
find_languages(src)Parse #nanopass.language annotations from a SourceLocRepr
find_languages_by_file(path)Parse language annotations from a file path
normalize_language_defs(langs)Validate and resolve entries; returns langs with resolved_entry populated
resolve(loc)Convert a SourceLoc to a structured SourceLocRepr
resolve_mbti(repr)Parse the .mbti interface for a package
NanoLangDef::diff(base, deriv)Compute the structural diff between two language definitions
NanoLangDef::with_constructors(self, constrs)Return a copy with a new constructor set (rebuilds production_defs)

Key types: NanoLangDef (with removed : Array[String] for #nano.remove), ProductionDef (with surface_forms, hooks), FormPattern, NanoHooks.

#@language — Generate AST scaffolding

APIDescription
Language::def(name?, loc~)Resolve languages from the callsite file
Language::from_file(name?, path~, mod~)Resolve languages from an explicit file
Language::gen()Generate AST definitions (Tree, wrappers, ext enums, terminals, entry stubs)
Language::layout()Compute GeneratedLanguageLayout (shared by gen and codec generator)
Language::expand()Expand the group into ExpandedLanguageGroup (inheritance + remove applied)
expand_language_group(raw)Standalone expansion without a Language object

GeneratedLanguageLayout carries ConstructorLayout per constructor: target (RawTarget / TreeTarget / ExtTarget), field_order, surface_forms, and hooks.

#@unparser/sexp — S-expression runtime

APIDescription
parse(src)Parse a StringView into a Sexp
match_form(sexp, pattern, field_count)Match a sexp against a $0/$1-placeholder pattern
form_doc(pattern, fields)Render field docs into a surface form
expect_atom / expect_list / expect_arityStructural decode helpers
hook_error / decode_error / literal_failureStructured error constructors
SexpDoc::render(self, width?)Pretty-print a doc to a string

Error hierarchy: SexpError { Syntax | Decode | HookError | GeneratedCode }. Cause-chained context via with_context.

#@unparser/gen — Codec generator

APIDescription
gen(lang)Generate Array[@syntax.Impl] codecs for all types in a language group
gen_source(lang)Return the generated source as a String (for inspection/snapshotting)

Generated per type: Ty::from_sexp, Ty::from_sexp_without_hooks (safe hook fallback), Ty::parse_sexp, Ty::to_sexp_doc, Ty::unparse_sexp.

#@pass — Transformation-pass scaffolding

See pass/README.mbt.md for the full API. Highlights:

MilestoneWhat is generated
M1 (surface)Smart constructors + view functions for wrapper-based ASTs
M2 (cata)Catamorphism cata for bottom-up folds
M3 (rewrite_m)Effectful delayed-subtree rewrite with RewriteAlg
M4 (stub)Diff-driven pass stub generator
M5 (PassM)Effectful pipeline/trace combinators

#Design Rationale

#Why code generation instead of a generic runtime?

The generated Tree has per-language-group type parameter lists. A generic runtime library cannot express typed operations over arbitrary tree shapes without higher-kinded types. Code generation produces exactly-typed syntax scaffolding per language group; the same layout decisions drive AST generation, S-expression codec generation, and pass scaffolding.

#How extra type variables work

When a derived language adds arguments to an inherited constructor (e.g., Lam gains a Type parameter in SimplyTypedLambda), the codegen introduces a fresh type variable (LamE). For base languages, it's filled with Unit. This keeps the Tree enum generic while preserving type safety per concrete language.

#Goals

#TODOs

  • #nano.form named placeholders$x / $body by meta-var name, repeat (...), optional, and dotted-list patterns are parsed by meta_parser but not yet consumed by the codec generator.
  • +/- extension syntax — explicit add/remove in the enum body (current #nano.remove covers top-level removal; inline +/- markers are not yet supported).
  • layout.group type convergenceGeneratedLanguageLayout.group is still Array[NanoLangDef]; migrating to ExpandedLanguageGroup would eliminate the parallel shared/own derivation in the layout builder.

#Continued improvements

  • Better codegen ergonomics — reduce raise noise in generated output, improve formatting, and add docstrings to generated methods.
  • Error handling in passes — allow passes to raise typed errors, with ergonomic error propagation across the pipeline.
  • Pass composition — a higher-level API for chaining passes with shared state, dependency tracking, and short-circuit on error.

#Incremental computation

Nanopass decomposes a compiler into dozens of small, single-purpose passes. Maybe this can be a natural fit for incremental computation: when a source file changes, only the passes whose inputs are affected need to re-run — the rest can reuse cached results.

However, how to achieve this remains to be explored.

We aim to explore:

  • Pass dependency tracking — each pass declares its input and output language; the framework can infer a dependency graph and determine the minimal set of passes to re-execute.
  • Granular caching — cache pass results per AST node or subtree. When a leaf changes, only the ancestor nodes on the path to the root need re-computation.
  • Dirtying and invalidation — integrate with editor tooling (LSP) to mark specific AST regions as dirty and incrementally re-run passes on just those regions.

The long-term vision is a compiler that responds to each keystroke with near-instant feedback — reusing as much prior work as possible, recomputing only what changed.

#References

[1] A. Keep. A Nanopass Framework for Commercial Compiler Development. Doctoral dissertation, Indiana University, Bloomington, Indiana, USA, Feb. 2013.

[2] S. Najd, S. Peyton Jones. Trees that Grow. Journal of Universal Computer Science, Vol. 23, No. 1, pp. 47-62, Jan. 2017.