🥮 mooncakes.io

/

/

#scanner

The lossless scanner, a pipeline of pure stages: scan_pieces unfolds scan_at over the text into steps (a token or a trivia piece each), attach_trivia groups them into tokens with their comments and whitespace, and tokenize is the two composed. The scan never fails: a problem becomes an Error token and a diagnostic, and TokenStream::to_source() is always the text, byte for byte.

///|
test {
  let s = @scanner.tokenize_text("string Foo\n")
  inspect(s.to_source(), content="string Foo\n")
  debug_inspect(
    s.tokens.iter().map(t => t.kind).collect(),
    content=(
      #|[Identifier, Identifier]
    ),
  )
  inspect(s.has_errors(), content="false")
}

Comment

pub(all) struct Comment {
  kind : CommentKind
  text : String
  span : @moonrockz/smithy-syntax/span.Span
Span
} derive(Eq, @moonbitlang/core/debug.Debug
Debug)

A comment. span covers the slashes and the text up to the line break (not included); text is what follows the slashes.

CommentKind

pub(all) enum CommentKind {
  Line
  Doc
} derive(Eq, Hash, @moonbitlang/core/debug.Debug
Debug)

// or ///.

Piece

pub(all) enum Piece {
  Tok(TokenKind, String, @moonrockz/smithy-syntax/span.Span
Span)
  Triv(Trivia)
} derive(Eq, @moonbitlang/core/debug.Debug
Debug)

What one scanning step produces, in source order.

Step

pub(all) struct Step {
  piece : Piece
  end : @moonrockz/smithy-syntax/span.Offset
Offset
  diagnostics : ArrayView[@moonrockz/smithy-syntax/diagnostic.Diagnostic
Diagnostic]
} derive(Eq, @moonbitlang/core/debug.Debug
Debug)

One step of the scan: the piece read at an offset, where the next step starts, and the lexical problems found in the piece (several for a string with several bad escapes).

Token

pub(all) struct Token {
  kind : TokenKind
  lexeme : String
  span : @moonrockz/smithy-syntax/span.Span
Span
  trivia_before : ArrayView[Trivia]
  trivia_after : ArrayView[Trivia]
} derive(Eq, @moonbitlang/core/debug.Debug
Debug)

A token with the trivia around it. Trivia after a token up to and including the first line break is its trivia_after; everything else is the next token's trivia_before.

Token::to_source

fn Token::to_source(self : Token) -> String

trivia_before, the lexeme and trivia_after, as in the source.

TokenKind

pub(all) enum TokenKind {
  Identifier
  Number
  String
  TextBlock
  At
  Colon
  Walrus
  Equal
  Dot
  Pound
  Dollar
  LBrace
  RBrace
  LBracket
  RBracket
  LParen
  RParen
  Error
} derive(Eq, Hash, @moonbitlang/core/debug.Debug
Debug)

The token kinds, the reference's IdlToken without its whitespace kinds (those are Trivia) and without EOF. Keywords are identifiers: Smithy keywords are contextual and the parser recognizes them.

TokenStream

pub(all) struct TokenStream {
  tokens : ArrayView[Token]
  trivia : ArrayView[Trivia]
  diagnostics : ArrayView[@moonrockz/smithy-syntax/diagnostic.Diagnostic
Diagnostic]
} derive(Eq, @moonbitlang/core/debug.Debug
Debug)

The result of a scan. trivia is the trivia of a text with no tokens, and empty otherwise. diagnostics are the lexical problems in order.

TokenStream::has_errors

fn TokenStream::has_errors(self : TokenStream) -> Bool

Whether any diagnostic has severity Error.

TokenStream::to_source

fn TokenStream::to_source(self : TokenStream) -> String

The scanned text, byte for byte.

Trivia

pub(all) enum Trivia {
  Whitespace(String, @moonrockz/smithy-syntax/span.Span
Span)
  Newline(String, @moonrockz/smithy-syntax/span.Span
Span)
  Comma(@moonrockz/smithy-syntax/span.Span
Span)
  Comment(Comment)
} derive(Eq, @moonbitlang/core/debug.Debug
Debug)

Text between tokens. The specification treats a comma as whitespace, so it is trivia here.

Trivia::is_comment

fn Trivia::is_comment(self : Trivia) -> Bool

Whether this trivia is a comment.

Trivia::is_newline

fn Trivia::is_newline(self : Trivia) -> Bool

Whether this trivia is a line break.

Trivia::span

fn Trivia::span(self : Trivia) -> @moonrockz/smithy-syntax/span.Span
Span

The span the trivia covers.

Trivia::text

fn Trivia::text(self : Trivia) -> String

The source text of the trivia.

attach_trivia

fn attach_trivia(steps : Array[Step]) -> TokenStream

Group the steps into tokens with trivia and collect the diagnostics. Trivia after a token up to and including the first line break is its trivia_after; the rest goes to the next token's trivia_before. Trailing trivia after the last token's line break stays with that token.

is_integer_lexeme

fn is_integer_lexeme(lexeme : String) -> Bool

Whether a Number lexeme has no fraction and no exponent.

scan_at

fn scan_at(text : String, at : @moonrockz/smithy-syntax/span.Offset
Offset) -> Step?

One token or one piece of trivia at at, or None at the end of the text. Pure: the same text and offset always give the same step.

scan_pieces

fn scan_pieces(text : String) -> Array[Step]

Every step of the text, in order: an unfold over scan_at. The only state is the offset local to this function.

text_block_value

fn text_block_value(lexeme : String) -> String

The value of a TextBlock token, per the specification: drop the opening quotes, optional spaces and the first line break; split the rest into lines; remove the common indent and trailing spaces; join with \n; then decode escapes.

tokenize

fn tokenize(source : @moonrockz/smithy-syntax/span.SourceText
SourceText) -> TokenStream

Scan source: a pipeline of pure stages. The scan never fails and is lossless: TokenStream::to_source() is the text.

tokenize_text

fn tokenize_text(text : String) -> TokenStream

tokenize of a SourceText built from text.

unescape

fn unescape(lexeme : String) -> String

The value of a String token: the lexeme with its quotes removed and its escapes decoded.

Source Files

Powered by MoonBit

Site source Report issue Packages Build queue Skills Statistics

© 2026 mooncakes.io