🥮 mooncakes.io

    #@tokenizer

    Low-level HTML5 tokenizer. Drives the higher-level tree builder in @parser, but is also useful on its own when you only need a flat stream of tags, text, comments, and doctypes — for example to count tag usage, strip a specific element, or build a custom syntax highlighter.

    The examples below are mbt check blocks and run as part of moon test tokenizer.

    #A first token stream

    tokenize returns a TokenizedHtml with a list of tokens. Every token stream ends with an explicit Eof.

    ///|
    test "readme tokenize hello" {
      let result = @tokenizer.tokenize("<p class='hi'>Hello</p>")
      debug_inspect(
        result.tokens,
        content=(
          #|[
          #|  Tag(
          #|    {
          #|      kind: StartTag,
          #|      name: "p",
          #|      attrs: { "class": Some("hi") },
          #|      self_closing: false,
          #|    },
          #|  ),
          #|  Characters("Hello"),
          #|  Tag({ kind: EndTag, name: "p", attrs: {}, self_closing: false }),
          #|  Eof,
          #|]
        ),
      )
    }

    #Token shape at a glance

    ///|
    pub(all) enum HtmlToken {
      Tag(TagToken)
      Characters(String)
      CommentToken(String)
      DoctypeToken(DoctypeInfo)
      Eof
    }

    • Tag carries kind (StartTag or EndTag), name, an attribute map, and the self_closing flag.
    • Characters is the decoded text — entities like &amp; are already expanded.
    • CommentToken holds the comment data without the <!--/--> delimiters.
    • DoctypeToken holds the name, optional public/system identifiers, and the force_quirks flag.

    #Collecting errors

    By default the tokenizer recovers silently. Pass collect_errors=true to get structured diagnostics alongside the recovered tokens.

    ///|
    test "readme tokenize collect errors" {
      let result = @tokenizer.tokenize("<div a=>", collect_errors=true)
      debug_inspect(
        result,
        content=(
          #|{
          #|  tokens: [
          #|    Tag(
          #|      {
          #|        kind: StartTag,
          #|        name: "div",
          #|        attrs: { "a": Some("") },
          #|        self_closing: false,
          #|      },
          #|    ),
          #|    Eof,
          #|  ],
          #|  errors: [
          #|    {
          #|      code: "missing-attribute-value",
          #|      line: Some(1),
          #|      column: Some(8),
          #|      category: "tokenizer",
          #|      message: "missing-attribute-value",
          #|    },
          #|  ],
          #|}
        ),
      )
    }

    #Decoding character references

    decode_entities exposes the same decoder the tokenizer uses on text nodes and attribute values, in case you already have the raw string.

    ///|
    test "readme decode entities" {
      let decoded = @tokenizer.decode_entities(
        "Tom &amp; Jerry &#x26; friends",
        in_attribute=false,
      )
      debug_inspect(
        decoded,
        content=(
          #|"Tom & Jerry & friends"
        ),
      )
    }

    #Streaming callback variant

    When you do not want the intermediate array, tokenize_each invokes a callback per token:

    ///|
    test "readme tokenize_each counts tags" {
      let mut start_tags = 0
      @tokenizer.tokenize_each("<p>one</p><p>two</p>", token => {
        match token {
          Tag({ kind: StartTag, .. }) => start_tags = start_tags + 1
          _ => ()
        }
      })
      assert_eq(start_tags, 2)
    }

    #When to reach for @parser instead

    The tokenizer is intentionally context-free: it does not insert implicit <html> / <head> / <body> elements, balance mismatched tags, or foster-parent table contents. For a real DOM with HTML5 recovery semantics, call @parser.parse (or the convenience wrappers at the crate root). Use @tokenizer directly when you want the raw token stream without that machinery.

    DoctypeInfo

    source
    pub(all) struct DoctypeInfo {
      name : String
      public_id : String?
      system_id : String?
      force_quirks : Bool
    } derive(Eq, 
    Debug    )

    Doctype data emitted by the tokenizer.

    force_quirks is set when the tokenizer sees a malformed or legacy doctype form that should place a parsed document into quirks mode.

    DoctypeInfo::equal

    source
    #deprecated("implicit derived-impl promotion; call the trait method directly")
    fn DoctypeInfo::equal(DoctypeInfo, DoctypeInfo) -> Bool

    DoctypeInfo::not_equal

    source
    #deprecated("implicit derived-impl promotion; call the trait method directly")
    fn DoctypeInfo::not_equal(x : DoctypeInfo, y : DoctypeInfo) -> Bool

    DoctypeInfo::to_repr

    source
    #deprecated("implicit derived-impl promotion; call the trait method directly")
    fn DoctypeInfo::to_repr(DoctypeInfo) -> 
    Repr

    HtmlToken

    source
    pub(all) enum HtmlToken {
      Tag(TagToken)
      Characters(String)
      CommentToken(String)
      DoctypeToken(DoctypeInfo)
      Eof
    } derive(Eq, 
    Debug    )

    Token variants produced by the HTML tokenizer.

    Eof is appended as the final token. Character data and comments are stored as already-normalized MoonBit strings; parse diagnostics are returned through TokenizedHtml.errors when error collection is enabled.

    HtmlToken::equal

    source
    #deprecated("implicit derived-impl promotion; call the trait method directly")
    fn HtmlToken::equal(HtmlToken, HtmlToken) -> Bool

    HtmlToken::not_equal

    source
    #deprecated("implicit derived-impl promotion; call the trait method directly")
    fn HtmlToken::not_equal(x : HtmlToken, y : HtmlToken) -> Bool

    HtmlToken::to_repr

    source
    #deprecated("implicit derived-impl promotion; call the trait method directly")
    fn HtmlToken::to_repr(HtmlToken) -> 
    Repr

    TagKind

    source
    pub(all) enum TagKind {
      StartTag
      EndTag
    } derive(Eq, 
    Debug    )

    Whether a tag token opens an element or closes one.

    StartTag represents tags such as <p> and EndTag represents tags such as </p>.

    TagKind::equal

    source
    #deprecated("implicit derived-impl promotion; call the trait method directly")
    fn TagKind::equal(TagKind, TagKind) -> Bool

    TagKind::not_equal

    source
    #deprecated("implicit derived-impl promotion; call the trait method directly")
    fn TagKind::not_equal(x : TagKind, y : TagKind) -> Bool

    TagKind::to_repr

    source
    #deprecated("implicit derived-impl promotion; call the trait method directly")
    fn TagKind::to_repr(TagKind) -> 
    Repr

    TagToken

    source
    pub(all) struct TagToken {
      kind : TagKind
      name : String
      attrs : Map[String, String?]
      self_closing : Bool
    } derive(Eq, 
    Debug    )

    A start or end tag emitted by tokenize.

    name is lower-cased for HTML tag tokens. attrs maps attribute names to optional values; a value of None represents a minimized or missing-value attribute. self_closing records the solidus marker on start tags.

    TagToken::equal

    source
    #deprecated("implicit derived-impl promotion; call the trait method directly")
    fn TagToken::equal(TagToken, TagToken) -> Bool

    TagToken::not_equal

    source
    #deprecated("implicit derived-impl promotion; call the trait method directly")
    fn TagToken::not_equal(x : TagToken, y : TagToken) -> Bool

    TagToken::to_repr

    source
    #deprecated("implicit derived-impl promotion; call the trait method directly")
    fn TagToken::to_repr(TagToken) -> 
    Repr

    TokenizedHtml

    source

    Result returned by tokenize.

    tokens always ends with HtmlToken::Eof. errors is empty unless collect_errors=true was passed.

    TokenizedHtml::to_repr

    source
    #deprecated("implicit derived-impl promotion; call the trait method directly")
    fn TokenizedHtml::to_repr(TokenizedHtml) -> 
    Repr

    decode_entities

    source
    fn decode_entities(input : StringView, in_attribute~ : Bool) -> String

    Decode HTML character references in input.

    tokenize

    source
    fn tokenize(html : StringView, collect_errors? : Bool, xml_coercion? : Bool) -> TokenizedHtml

    Tokenize an HTML string without building a DOM tree.

    Set collect_errors=true to collect tokenizer diagnostics in the returned TokenizedHtml. Set xml_coercion=true to replace XML-invalid text and comment characters during tokenization.

    tokenize_each

    source
    fn tokenize_each(html : StringView, emit : (HtmlToken) -> Unit, xml_coercion? : Bool) -> Unit

    Tokenize an HTML string and emit tokens incrementally.

    The callback receives tokenizer output in source order, without an explicit Eof token.

    Source Files

    Powered by MoonBit

    Site sourceReport issuePackagesBuild queueSkillsStatistics

    © 2026 mooncakes.io