🥮 mooncakes.io

    html_parser

    MoonBit HTML parser and sanitizer ported from JustHTML.

    html
    parser
    sanitizer
    dom
    markdown
    Download zip
    Author
    Version
    0.1.8
    License
    Apache-2.0
    Repository
    Last updated
    10 days ago
    Downloads
    6K

    Dependencies

    #bobzhang/html_parser

    MoonBit port of the JustHTML parser, ported from the Python reference implementation in .repos/justhtml.

    The port currently provides:

    • Public DOM node builders.
    • Compact and simple pretty HTML serialization.
    • Text extraction.
    • Tokenization, document parsing, fragment parsing, and HTML5-style recovery coverage from vendored tokenizer/tree-builder fixtures.
    • Document-mode scaffolding with html/head/body, table/foreign-content recovery, template handling, source locations, and strict-mode errors.
    • CSS-style DOM queries for tag, class, id, attributes, combinators, selector lists, and common pseudo selectors.
    • Byte input parsing with supported transport labels, BOM sniffing, and meta-charset prescan.
    • Default-policy DOM sanitization with tag/attribute allowlists, comment and doctype handling, unsafe attribute filtering, basic URL checks, and explicit parse-time sanitization via sanitize=true. Custom policies can unwrap, drop, or escape disallowed tags, force hardened anchor rel tokens, and allowlist simple inline CSS properties while stripping invisible Unicode and hardening allowed raw-text, foreign text-integration, active foreign contents, foreign URL-function attributes, meta refresh contents, and base URL rewrites. URL-like attributes require exact URL policy rules. Policies can keep the default strip behavior, collect security findings, or raise on the first unsafe construct, and can define exact tag/attribute URL rules for schemes, hosts, fragments, relative URLs, protocol-relative rewrites, and allow/strip handling while rejecting malformed host values and backslashes. URL handling can also rewrite or drop URLs through UrlFilter, then proxy validated URLs through a policy-level or per-rule UrlProxy, including single URL attributes, simple srcset/imagesrcset candidate lists, ping/attributionsrc URL-token lists, and plain CSS url(...) values on allowlisted inline-style properties. Use css_preset_text() for the conservative text-style property allowlist from the reference sanitizer.
    • Transform helpers for sanitize, drop/unwrap/escape, pruning, linkification, attribute edits, and transform observers.
    • Linkify, streaming parse events, Markdown conversion, and an embeddable CLI runner plus native CLI wrapper.

    #Install

    moon add bobzhang/html_parser

    Then import the package from the moon.pkg that uses it:

    ///|
    import {
      "bobzhang/html_parser",
    }

    #Library Examples

    The examples below are mbt check doctests and are run by moon test.

    ///|
    test "readme parse fragment example" {
      let doc = @html_parser.parse_fragment(
        "<p class='intro'>Hello <b>MoonBit</b></p>",
      )
      assert_eq(
        doc.to_html(pretty=false),
        "<p class=\"intro\">Hello <b>MoonBit</b></p>",
      )
      assert_eq(doc.to_text(separator="", strip=false), "Hello MoonBit")
      assert_eq(doc.to_markdown(), "Hello **MoonBit**")
      assert_eq(doc.query("p.intro").length(), 1)
      let tokens = @html_parser.tokenize("<p>Hello</p>").tokens
      assert_eq(tokens.length(), 4)
    }

    ///|
    test "readme parse bytes example" {
      let bytes = @utf8.encode("<meta charset=utf-8><p>\u{20AC}</p>")
      let doc = @html_parser.parse_bytes(bytes)
      guard doc.encoding is Some("utf-8") else { fail("expected utf-8 encoding") }
      assert_eq(doc.to_text(separator="", strip=false), "\u{20AC}")
    }

    ///|
    test "readme sanitize dom example" {
      let doc = @html_parser.parse(
        "<!DOCTYPE html><!--x--><p onclick=alert(1)>ok</p><script>alert(1)</script>",
        sanitize=false,
      )
      let clean = @html_parser.sanitize_dom(
        doc.root,
        policy=@html_parser.default_sanitization_policy(),
      )
      assert_eq(@html_parser.to_html(clean, pretty=false), "<p>ok</p>")
      let fragment = @html_parser.parse_fragment(
        "<p onclick=alert(1)>ok</p><script>alert(1)</script>",
        sanitize=true,
      )
      assert_eq(fragment.to_html(pretty=false), "<p>ok</p>")
    }

    ///|
    test "readme CLI reader example" {
      let paths : Array[String] = []
      let result = @html_parser.run_cli_with_reader(["-", "--format", "text"], fn(
        path,
      ) {
        paths.push(path)
        @utf8.encode("<p>Hello <b>MoonBit</b></p>")
      })
      @test.assert_eq(paths, ["-"])
      assert_eq(result.exit_code, 0)
      assert_eq(result.stdout, "Hello MoonBit\n")
    }

    #Per-package documentation

    Each package ships its own README.mbt.md with runnable examples (mbt check blocks executed by moon test). They cover the package's public API surface and double as snapshot tests via debug_inspect.

    PackageReadsWhen you want to
    @tokenizerbytes → tokenswalk the raw HTML5 token stream
    @parsertokens → DOMbuild a real DOM tree with HTML5 recovery
    @streambytes → eventsdrive a streaming consumer (serializer, linter)
    @dom—construct or mutate DOM nodes directly
    @selectorDOM + CSSrun CSS-style queries against a tree
    @serializerDOM → bytesserialize back to compact or pretty HTML, or extract text
    @sanitizeDOM → DOMstrip unsafe content with allowlist policies
    @transformDOM → DOMrun a composable rewrite pipeline
    @linkifytext/DOM → DOMauto-link URLs and emails
    @markdownDOM → stringconvert a tree to Markdown source
    @cliargv → resultembed the justhtml runner in another tool
    @core—match on HtmlError / inspect ParseError

    #Native CLI

    The native CLI wrapper lives in cmd/justhtml and uses moonbitlang/async for raw stdin/file IO, stdout/stderr, and output files without custom C stubs. Build it from this repository with:

    moon run --target native --release --build-only cmd/justhtml

    The executable is written to _build/native/release/build/bobzhang/html_parser/cmd/justhtml/justhtml.exe. For example:

    printf '<p>Hello <b>MoonBit</b></p>' \
  | _build/native/release/build/bobzhang/html_parser/cmd/justhtml/justhtml.exe - --format text

    Black-box CLI integration tests live in tests/cram and run with:

    moon cram test --release tests/cram

    This builds the native executables and runs the cram documents with their build directories on PATH.

    #Workspace Examples

    This repository also has a moon.work workspace with an examples module for runnable documentation. The formatter example uses the local parser checkout:

    moon run --target native examples/cmd/htmlfmt -- "<article><p>Hello <b>MoonBit</b></p></article>"

    The examples documentation lives in examples/README.mbt.md.

    #Benchmarks

    Benchmark the main entry points (parse, sanitize, serialize, selector query, Markdown conversion, linkify, streaming) against the vendored Wikipedia portal page (~300 KB UTF-16):

    moon run --target native --release cmd/benchmark

    Pass a path to benchmark a different document, --json for machine-readable per-workload summaries (one JSON object per line), or --quick for a single smoke iteration per workload. CI builds and smoke-runs the benchmark on every push so the workloads cannot bit-rot; timing numbers are informational and never gate CI.

    #Property-Based Tests

    Alongside the example-based suite, quickcheck_*_test.mbt in the root package runs moonbitlang/core/quickcheck properties over generated input. Two generators cover different ground:

    • HtmlSource produces deliberate token soup — mis-nested tags, unmatched end tags, raw text, foreign content, malformed markup declarations — and drives the properties that must hold for any input: entry points never raise, parse_bytes agrees with parse, every node query returns matches its selector, and sanitized output is free of unsafe elements, on* attributes and javascript: URLs.
    • HtmlDocument produces well-nested documents that respect HTML's content models, and drives the round-trip properties. That distinction matters: serializing arbitrary soup is legitimately not round-trippable (a <p> inside a <p>, foster-parented table content, or <plaintext> all reparse differently), so a fixpoint assertion only makes sense on well-nested input.
    • HandBuiltDom builds trees through the public @dom constructors rather than through parse, so it can produce what the tree builder never would: a foreign ancestor over an HTML element, a name that contradicts its namespace, an element nested inside a textarea. Those shapes are where serialization has historically gone wrong, and the properties over them assert one thing — text carrying <img src=x onerror=1> must not come back as an element, whichever node is serialized, at either pretty setting, reparsed in either scripting mode.

    The generator is the bug-finding surface, so each of these properties is checked by mutation: reverting the serializer to an earlier, known-wrong rule must falsify it. A property that cannot fail looks exactly like one that passes.

    Seeds are fixed, so failures reproduce exactly. Counterexamples are shrunk against the generator's structure and reported as HTML source.

    #Development Checks

    Run the same validation entrypoint used by CI:

    moon run --target native scripts/check_ci.mbtx --skip-without-credentials

    Drop --skip-without-credentials when logged in locally and checking the full Mooncakes dry-run path. The script checks release-version consistency, MoonBit script inventory and argument smoke paths, validation-inventory wiring, GitHub workflow drift and tracked workflow inventory including the Copilot setup workflow, source layout, test-name inventory, migration docs, local Git hook wiring, vendored fixture sync when .repos/justhtml is present, vendored fixture manifest hashes, package metadata, formatting, generated interfaces, all supported targets, default/JS/native tests with a count floor, coverage, native CLI smoke behavior, Moon Cram CLI integration tests, Mooncakes package validation, and dynamic Mooncakes archive inventory/content checks.

    CliReadPlan

    source

    Describes how the CLI should obtain input after argument parsing.

    CliImmediate means parsing already produced a complete result, such as help, version output, or an argument error. CliReadPath asks the caller to read the given path, where "-" conventionally means standard input.

    CliResult

    source

    Result produced by the embeddable CLI runner.

    stdout and stderr contain terminal output. When file_output_path is present, file_output_content contains the bytes that a shell wrapper should write to that path instead of printing to standard output.

    DecideAction

    source

    Action returned by a TransformSpec::decide callback.

    Keep leaves the matched node unchanged. The other variants apply the same structural operations as the corresponding selector transforms.

    DisallowedTagHandling

    source

    How sanitizer handles elements whose tag names are not allowlisted.

    DoctypeInfo

    source

    Doctype data emitted by the tokenizer.

    force_quirks is set when the tokenizer sees a malformed or legacy doctype form that should place a parsed document into quirks mode.

    FragmentContext

    source

    Context element used when parsing an HTML fragment.

    The tag name is normalized to lowercase. ns can be used for foreign content contexts such as SVG or MathML.

    HtmlContext

    source

    Output context used by HTML serialization.

    Non-Html contexts escape serialized output for embedding in JavaScript strings, HTML attribute values, or URL text.

    HtmlError

    source

    Error type raised by strict parsing, serialization, sanitizer, and selector operations.

    HtmlToken

    source

    Token variants produced by the HTML tokenizer.

    Eof is appended as the final token. Character data and comments are stored as already-normalized MoonBit strings; parse diagnostics are returned through TokenizedHtml.errors when error collection is enabled.

    LinkMatch

    source

    A URL or email-like span found in plain text.

    start and end are UTF-16 offsets into the input StringView. kind is "url" or "email".

    LinkifyConfig

    source

    Options for the plain-text link scanner.

    Node

    source

    DOM node used for documents, fragments, elements, text, comments, and doctypes.

    Nodes are created with helpers such as document, fragment, element, text, comment, and doctype.

    NodeKind

    source

    Kind of DOM node represented by Node.

    ParseError

    source

    Parser or tokenizer diagnostic with optional source location.

    category identifies the source of the error, such as tokenizer or treebuilder. message defaults to code when no custom message is given.

    ParsedHtml

    source

    Result of parsing HTML.

    root is the document or fragment root. errors is populated when collect_errors=true or strict parsing observes an error. encoding is set by byte parsing APIs.

    SanitizationPolicy

    source

    DOM sanitization policy.

    A policy controls allowed tags and attributes, URL filtering, comment and doctype handling, foreign-content hardening, CSS style allowlists, selector limits used by transform hooks, and unsafe-input reporting.

    SanitizeTransformObserver

    source

    Observer callbacks for sanitizer-driven DOM rewrites.

    The node hook runs for events that have an associated DOM node. The report callback receives every sanitizer event message and the optional related node, including unsafe input that was stripped or collected.

    SelectorLimits

    source

    Resource limits used while parsing and matching CSS selectors.

    Limits are defensive bounds for selector length, nesting, list size, and match cost. Negative match budgets are treated as exhausted.

    StreamDoctypeEvent

    source

    Doctype event emitted by the streaming tokenizer facade.

    StreamEvent

    source

    Token-level streaming event.

    The stream API does not build a DOM tree. It forwards start tags, end tags, text, comments, and doctypes in tokenizer order, coalescing adjacent text.

    StreamStartEvent

    source

    Start-tag event emitted by the streaming tokenizer facade.

    name is the normalized tag name and attrs contains the decoded attributes for the tag.

    TagKind

    source

    Whether a tag token opens an element or closes one.

    StartTag represents tags such as <p> and EndTag represents tags such as </p>.

    TagToken

    source

    A start or end tag emitted by tokenize.

    name is lower-cased for HTML tag tokens. attrs maps attribute names to optional values; a value of None represents a minimized or missing-value attribute. self_closing records the solidus marker on start tags.

    TokenizedHtml

    source

    Result returned by tokenize.

    tokens always ends with HtmlToken::Eof. errors is empty unless collect_errors=true was passed.

    TransformSpec

    source

    A DOM transform specification for apply_transforms.

    The current port covers deterministic structural, attribute, callback, URL/style, utility, and sanitizer transforms. Most selector and node-kind transforms also support hook/report callbacks.

    UnsafeHandling

    source

    How sanitizer reports unsafe input that it strips or rewrites.

    UrlFilter

    source

    Callback wrapper used to rewrite or reject URL values before validation.

    The callback receives normalized tag name, normalized attribute name, and the raw attribute value. Returning None drops the URL.

    UrlHandling

    source

    Action used after a URL value passes the configured URL checks.

    UrlPolicy

    source

    URL sanitization policy shared by URL-bearing attributes.

    Exact (tag, attr) rules take precedence. Unmatched URL-like attributes use the default handling and relative-URL behavior.

    UrlPolicyRule

    source

    URL rule bound to a tag and attribute name.

    UrlProxy

    source

    Proxy endpoint used when a URL rule selects UrlProxy.

    Sanitized URLs are emitted as url?param=<encoded-url> or url&param=<encoded-url> depending on whether the proxy URL already has a query string.

    UrlRule

    source

    Per-attribute URL validation rule.

    Rules can restrict schemes and hosts, disallow fragments, normalize protocol-relative URLs, override relative-URL handling, or route accepted URLs through a proxy.

    apply_transforms

    source

    Apply DOM transforms in order.

    The transform mutates and returns the current root. Selector transforms process the children of the supplied root, matching the Python constructor-time pipeline where the root is normally a document or document fragment container.

    cli_help

    source
    fn cli_help() -> String

    Return the CLI usage text.

    cli_read_plan

    source

    Parse CLI arguments and report whether the caller must read an input path.

    Use this when integrating with an environment that performs its own file or standard-input IO. For direct byte input, use run_cli_bytes.

    comment

    source
    fn comment(data : StringView) -> 
    Node

    Create a comment node.

    css_preset_text

    source
    fn css_preset_text() -> Array[String]

    Return the conservative text-style CSS property allowlist.

    default_document_sanitization_policy

    source
    fn default_document_sanitization_policy() -> 
    SanitizationPolicy

    Return the default document sanitizer policy.

    This extends the fragment policy with document shell tags and preserves the doctype.

    default_sanitization_policy

    source

    Return the default fragment sanitizer policy.

    doctype

    source
    fn doctype(name? : String, public_id? : String, system_id? : String, force_quirks? : Bool) -> 
    Node

    Create a doctype node.

    document

    source

    Create a document node with optional children.

    element

    source
    fn element(name : StringView, attrs? : Map[String, String?], children? : Array[
    Node    ], ns? : String) -> 
    Node

    Create an element node.

    Namespace aliases html, svg, and mathml are normalized for serializer and sanitizer behavior. Child nodes are attached in order.
    fn find_links(text : StringView) -> Array[
    LinkMatch    ]

    Find URL and email-like spans in plain text using the default configuration.

    Find URL and email-like spans in plain text using an explicit configuration.

    fragment

    source

    Create a document-fragment node with optional children.

    linkify_dom

    source

    Linkify URL and email text inside a DOM subtree.

    The transform mutates and returns node. By default it skips existing anchors and whitespace-preserving tags (code, pre, script, style, and textarea), while still processing normal children such as template contents.

    matches

    source
    fn matches(node : 
    Node    , selector : StringView) -> Bool

    Return whether node itself matches a CSS selector.

    parse

    source
    fn parse(html : StringView, sanitize? : Bool, collect_errors? : Bool, strict? : Bool, scripting_enabled? : Bool, xml_coercion? : Bool, track_node_locations? : Bool, iframe_srcdoc? : Bool) -> 
    ParsedHtml     raise 
    HtmlError

    Parse a full HTML document from a string.

    The returned root is a document node with the usual html, head, and body scaffolding. Set sanitize=true to apply the default document sanitizer after parsing, collect_errors=true to keep parse diagnostics, and strict=true to raise @core.HtmlError::StrictMode on the first parse error.

    parse_bytes

    source
    fn parse_bytes(input : BytesView, encoding? : String, sanitize? : Bool, collect_errors? : Bool, strict? : Bool, scripting_enabled? : Bool, xml_coercion? : Bool, track_node_locations? : Bool, iframe_srcdoc? : Bool) -> 
    ParsedHtml     raise 
    HtmlError

    Decode and parse an HTML byte stream.

    When encoding is absent, BOMs and <meta charset> declarations are sniffed before falling back to Windows-1252. The detected or requested encoding is stored in ParsedHtml.encoding.

    parse_fragment

    source
    fn parse_fragment(html : StringView, context? : 
    FragmentContext    , sanitize? : Bool, collect_errors? : Bool, strict? : Bool, scripting_enabled? : Bool, xml_coercion? : Bool, track_node_locations? : Bool, iframe_srcdoc? : Bool) -> 
    ParsedHtml     raise 
    HtmlError

    Parse an HTML fragment from a string.

    context controls the fragment context element used by the tree builder. Without a context, the fragment is parsed into a generic fragment root. sanitize, collect_errors, strict, scripting_enabled, xml_coercion, and track_node_locations have the same meaning as in parse.

    query

    source

    Return all descendants of root that match a CSS selector.

    query_one

    source
    fn query_one(root : 
    Node    , selector : StringView) -> 
    Node    ?

    Return the first descendant of root that matches a CSS selector.

    run_cli_bytes

    source
    fn run_cli_bytes(args : ArrayView[String], input : BytesView) -> 
    CliResult

    Run the JustHTML-compatible CLI over already-read input bytes.

    args should contain only command-line arguments after the executable name. The returned CliResult captures exit status, terminal output, and optional output-file content for the caller to write.

    run_cli_with_reader

    source
    fn run_cli_with_reader(args : ArrayView[String], read_input : (String) -> Bytes) -> 
    CliResult

    Run the CLI with caller-provided path reading.

    The callback receives the input path selected by argument parsing. This is useful for native or embedded hosts that want the library to handle CLI options while the host owns filesystem or standard-input IO.

    sanitize_dom

    source

    Sanitize a DOM node in place and return the sanitized root.

    When no policy is supplied, document roots use the document policy and other roots use the fragment policy.

    stream

    source

    Parse an HTML string into streaming events.

    stream_bytes

    source
    fn stream_bytes(input : BytesView, encoding? : String) -> Array[
    StreamEvent    ]

    Decode HTML bytes and return streaming events.

    When encoding is omitted, the same byte-sniffing path used by parse_bytes chooses the input encoding.

    stream_bytes_each

    source
    fn stream_bytes_each(input : BytesView, emit : (
    StreamEvent    ) -> Unit, encoding? : String) -> Unit

    Decode HTML bytes and emit streaming events incrementally.

    When encoding is omitted, the same byte-sniffing path used by parse_bytes chooses the input encoding.

    stream_each

    source
    fn stream_each(html : StringView, emit : (
    StreamEvent    ) -> Unit) -> Unit

    Parse an HTML string and emit streaming events incrementally.

    text

    source
    fn text(data : StringView) -> 
    Node

    Create a text node.

    to_html

    source
    fn to_html(node : 
    Node    , pretty? : Bool, indent_size? : Int, context? : 
    HtmlContext    , quote? : Char) -> String raise 
    HtmlError

    Serialize a node as HTML.

    to_markdown

    source
    fn to_markdown(node : 
    Node    , html_passthrough? : Bool) -> String raise 
    HtmlError

    Render a DOM node and its descendants as Markdown.

    to_test_format

    source
    fn to_test_format(node : 
    Node    ) -> String

    Render a deterministic tree dump intended for conformance tests.

    to_text

    source
    fn to_text(node : 
    Node    , separator? : String, strip? : Bool, separator_blocks_only? : Bool) -> String

    Extract descendant text from a node.

    tokenize

    source
    fn tokenize(html : StringView, collect_errors? : Bool, xml_coercion? : Bool) -> 
    TokenizedHtml

    Tokenize an HTML string without building a DOM tree.

    Set collect_errors=true to collect tokenizer diagnostics in the returned TokenizedHtml. Set xml_coercion=true to replace XML-invalid text and comment characters during tokenization.

    Source Files

    Powered by MoonBit

    Site sourceReport issuePackagesBuild queueSkillsStatistics

    © 2026 mooncakes.io