🥮 mooncakes.io

    moonbitlang/core/lexbuf does not have a README file

    AsyncLexbuf

    source
    pub struct AsyncLexbuf {
      // private fields
    }

    An asynchronous UTF-16 input buffer for lexscan, fed by string chunks. Chunks may be released after lexscan has finished consuming them.

    AsyncLexbuf::from_fn

    source
    fn AsyncLexbuf::from_fn(source : async () -> String?) -> AsyncLexbuf

    Creates an initially empty lexbuf. source asynchronously returns the next chunk, or None at EOF. Empty chunks are ignored.

    Lexbuf

    source
    pub struct Lexbuf {
      // private fields
    }

    A synchronous UTF-16 input buffer for lexscan, fed by string chunks. Chunks may be released after lexscan has finished consuming them.

    Lexbuf::from_fn

    source
    fn Lexbuf::from_fn(source : () -> String?) -> Lexbuf

    Creates an initially empty lexbuf. source returns the next chunk, or None at EOF. Empty chunks are ignored.

    Lexbuf::from_string

    source
    #deprecated("Use `StringScanner` instead")
    fn Lexbuf::from_string(content : String) -> Lexbuf

    Creates a lexbuf whose complete input is already available.

    Deprecated: use StringScanner instead.

    StringScanner

    source
    pub(all) struct StringScanner {
      data : StringView
      cursor : Int
    }

    A lightweight scanner for a StringView input.

    StringScanner is a non-streaming target for lexscan. Unlike Lexbuf and AsyncLexbuf, it does not refill its input and does not support retention. The generated scanner advances cursor after each selected case, so the same value can be passed to successive lexscan expressions.

    Both data offsets and cursor are measured in UTF-16 code units. The cursor is relative to data and in the range 0..=data.length(). A StringView slice is therefore a valid input, and scanning it does not read outside the slice.

    Create a scanner with StringScanner(data). It checks that the cursor is in range and not inside a surrogate pair, and lexscan only moves the cursor to the end of a match, so the cursor stays at a character boundary.

    let scanner = @lexbuf.StringScanner("hello 42")

    let token = lexscan scanner {
      re"^[a-z]+" as word => word
      re"^ +" => " "
      _ => "?"
    }

    StringScanner::StringScanner

    source
    fn StringScanner::StringScanner(data : StringView, cursor? : Int) -> StringScanner

    Creates a scanner over data that starts at cursor, a UTF-16 code-unit offset relative to data (0 by default).

    Panics

    • If cursor is outside 0..=data.length().
    • If cursor is inside a surrogate pair of data.

    Example

    test {
      let scanner = @lexbuf.StringScanner("ab\u{1F600}c", cursor=2)
      inspect(scanner.cursor, content="2")
    }

    Source Files