mooncakes.io

marianoguerra/shrubbery/lexer does not have a README file

AtMode

pub(all) enum AtMode {
  Initial
  Args
  NoArgs
  OpContinue
  Open
  Inside
  Escape
  Close
} derive(Eq)

Where an @ form is in its own little grammar.

The names are the reference's. Initial through OpContinue are the part before the braces, where ordinary shrubbery tokens are read and the mode only decides what may come next; Open, Inside, Escape and Close are the braced text, where the scanner is reading characters rather than tokens.

LexMode

source

pub(all) enum LexMode {
  Initial
  Continuing
} derive(Eq)

The two lexer states.

They differ in the number pattern and in nothing else: Initial reads +2 as a signed literal, Continuing — entered after an identifier, a literal, a closer or a keyword — reads it as + applied to 2. That single bit is the whole of why 1+2 is addition and 1 +2 is two terms.

Token

source

pub(all) struct Token {
  kind : TokenKind
  text : String
  raw : String?
  value : @marianoguerra/shrubbery/sexp.Datum
Datum?
  span : @marianoguerra/shrubbery/basic.Span
Span
  partner : String?
}

A token, and the exact text it was made of.

Every code unit of the input belongs to exactly one token's text. Concatenating them reproduces the source byte for byte, whitespace and comments included. That total coverage is not a nicety — it is what makes the raw-text metadata, and therefore round-tripping, possible at all, so it has its own test.

Token::column

source

fn Token::column(self : Token) -> @marianoguerra/shrubbery/column.Column
Column

The indentation column this token takes part in comparisons at.

Not simply span.start.col: a | counts as half a column further right than where it sits. That convention is what lets a block's content and an alternative's | be at the same visual column and still be ordered against each other.

Token::end_line

source

fn Token::end_line(self : Token) -> Int

Token::is_trivia

source

fn Token::is_trivia(self : Token) -> Bool

Whether this token is whitespace or a comment — the things the parser skips but the raw-text metadata keeps.

Token::line

source

fn Token::line(self : Token) -> Int

Token::name

source

fn Token::name(self : Token) -> String

The name this token denotes: the escape's symbol where there is one, and the source text otherwise.

Token::raw_text

source

fn Token::raw_text(self : Token) -> String

The text the raw-text metadata should carry for this token.

TokenKind

source

pub(all) enum TokenKind {
  Identifier
  Keyword
  Literal(@marianoguerra/shrubbery/sexp.Datum
Datum)
  Operator
  BlockOperator
  ContinueOperator
  BarOperator
  Opener
  Closer
  CommaOperator
  SemicolonOperator
  Comment
  Whitespace
  GroupComment
  SExp(@marianoguerra/shrubbery/sexp.Datum
Datum)
  At
  AtOpener
  AtContent
  AtCloser
  AtComment
  SQuote
  EndOfInput
  Fail(@marianoguerra/shrubbery/error.ErrorKind
ErrorKind)
} derive(Eq)

What a token is.

The names are the reference's own, because the parser is written against them and a rename here would be a rename in two places that must agree.

TokenKind::kind_name

source

fn TokenKind::kind_name(self : TokenKind) -> String

The reference's name for this token kind.

Used by the token-parity oracle, so these strings are the reference's symbols and not ours to tidy.

Variant

source

pub(all) struct Variant {
  allow_operator : (String) -> Bool
  indented_operator_continue : (String) -> Bool
}

The two predicates that are the entire configurability of the notation.

allow_operator rejects an operator spelling outright; a variant that forbids <- makes it a read error rather than an operator. indented_operator_continue decides whether an indented line starting with that operator continues the previous group.

default_variant

source

let default_variant : Variant

lex_all

source

fn lex_all(src : String, variant? : Variant, start_column? : @marianoguerra/shrubbery/column.Column
Column) -> Array[Token]

Scan the whole source.

The token stream is produced in full before parsing starts. The reference interleaves them, but only because its lexer doubles as an editor's incremental colourer; nothing in the grouping layer feeds back into tokenisation, so there is nothing to interleave for.

rewrite_quotes

source

fn rewrite_quotes(tokens : Array[Token]) -> Array[Token]

Decide, for each ', whether it opens or closes.

' is its own opener and closer, so nesting is ambiguous on its face and the decision needs context. The rule: a ' outside quotes opens; inside quotes it closes, UNLESS a bracket has been opened since the last ', in which case it opens a nested quote. That is what makes 'a ('nested') b' read the way it looks, while two consecutive ' with nothing between them would otherwise be an opener and its own closer.

A separate pass rather than lexer state, so the scanner stays context-free and this rule can be read and tested on its own.