🥮 mooncakes.io

    marianoguerra/html/token does not have a README file

    RawAttr

    source
    pub(all) struct RawAttr {
      name : String
      value : String?
      raw : String?
      span : 
    Span
    } derive(Eq, 
    Debug    )

    An attribute as written, before the parser knows its namespace.

    name is ASCII-lowercased, which is right for HTML and wrong for SVG -- viewBox arrives here as viewbox. Restoring it needs the namespace, which needs the element stack, which is the parser's. Doing it here would mean lowercasing and then un-lowercasing, or not lowercasing and getting HTML wrong; doing it there is one table lookup at the point where the answer is known.

    Scanned

    source

    Everything noticed while scanning, in source order.

    The tokenizer collects rather than raises for the reason the CSS one does: a malformed character reference is a warning about six characters, and making it stop the scan would mean a page with one &copy in it produced no tokens at all.

    Start

    source
    pub(all) enum Start {
      Markup
      Escapable(String)
      Raw(String)
    } derive(Eq, 
    Debug    )

    Where a scan begins.

    Ordinary markup, almost always. The other two exist because the content model of an element is decided by the element that opened it, and a scan can begin already INSIDE one -- innerHTML on a <title>, a fragment handed to a template engine, a conformance suite naming an initial state. The tag is carried rather than the state, because what ends a raw-text run is that element's own end tag and nothing else.

    Token

    source

    Token::is_whitespace

    source
    fn Token::is_whitespace(self : Token) -> Bool

    Whether this token is whitespace-only character data.

    Asked by the parser at exactly one place -- deciding whether text before <html> is worth keeping -- and by tests. A method rather than a free function so that it reads as a property of the token.

    TokenKind

    source
    pub(all) enum TokenKind {
      StartTag(String, Array[RawAttr], Bool)
      EndTag(String)
      Text(String, String?)
      RawText(String)
      Comment(String)
      Doctype(String, String?, String?, Bool)
      Cdata(String)
      Bogus(
    ErrorKind    , String)
      Eof
    } derive(Eq, 
    Debug    )

    TokenKind::to_debug

    source
    fn TokenKind::to_debug(self : TokenKind) -> String

    A one-line rendering of a token, for the harness and for tests.

    dump

    source
    fn dump(tokens : Array[Token]) -> String

    A whole stream, one token per line.

    tokenize

    source
    fn tokenize(src : String, start? : Start) -> Scanned

    Tokenize a whole document, ending with exactly one Eof.

    tokens

    source
    fn tokens(src : String, start? : Start) -> Array[Token]

    Tokenize, keeping only the tokens.

    Source Files

    Powered by MoonBit

    Site sourceReport issuePackagesBuild queueSkillsStatistics

    © 2026 mooncakes.io