🥮 mooncakes.io

html_parser

MoonBit HTML parser and sanitizer ported from JustHTML.

html
parser
sanitizer
dom
markdown
moon add bobzhang/html_parser@0.1.7
Download zip
Author
Version
0.1.7
License
Apache-2.0
Repository
Last updated
2 months ago
Downloads
3K

Dependencies

README

#bobzhang/html_parser

MoonBit port of the JustHTML parser, ported from the Python reference implementation in .repos/justhtml.

The port currently provides:

  • Public DOM node builders.
  • Compact and simple pretty HTML serialization.
  • Text extraction.
  • Tokenization, document parsing, fragment parsing, and HTML5-style recovery coverage from vendored tokenizer/tree-builder fixtures.
  • Document-mode scaffolding with html/head/body, table/foreign-content recovery, template handling, source locations, and strict-mode errors.
  • CSS-style DOM queries for tag, class, id, attributes, combinators, selector lists, and common pseudo selectors.
  • Byte input parsing with supported transport labels, BOM sniffing, and meta-charset prescan.
  • Default-policy DOM sanitization with tag/attribute allowlists, comment and doctype handling, unsafe attribute filtering, basic URL checks, and explicit parse-time sanitization via sanitize=true. Custom policies can unwrap, drop, or escape disallowed tags, force hardened anchor rel tokens, and allowlist simple inline CSS properties while stripping invisible Unicode and hardening allowed raw-text, foreign text-integration, active foreign contents, foreign URL-function attributes, meta refresh contents, and base URL rewrites. URL-like attributes require exact URL policy rules. Policies can keep the default strip behavior, collect security findings, or raise on the first unsafe construct, and can define exact tag/attribute URL rules for schemes, hosts, fragments, relative URLs, protocol-relative rewrites, and allow/strip handling while rejecting malformed host values and backslashes. URL handling can also rewrite or drop URLs through UrlFilter, then proxy validated URLs through a policy-level or per-rule UrlProxy, including single URL attributes, simple srcset/imagesrcset candidate lists, ping/attributionsrc URL-token lists, and plain CSS url(...) values on allowlisted inline-style properties. Use css_preset_text() for the conservative text-style property allowlist from the reference sanitizer.
  • Transform helpers for sanitize, drop/unwrap/escape, pruning, linkification, attribute edits, and transform observers.
  • Linkify, streaming parse events, Markdown conversion, and an embeddable CLI runner plus native CLI wrapper.

#Install

moon add bobzhang/html_parser

Then import the package from the moon.pkg that uses it:

import {
  "bobzhang/html_parser"
}

#Library Examples

The examples below are mbt check doctests and are run by moon test.

///|
test "readme parse fragment example" {
  let doc = @html_parser.parse_fragment(
    "<p class='intro'>Hello <b>MoonBit</b></p>",
  )
  assert_eq(
    doc.to_html(pretty=false),
    "<p class=\"intro\">Hello <b>MoonBit</b></p>",
  )
  assert_eq(doc.to_text(separator="", strip=false), "Hello MoonBit")
  assert_eq(doc.to_markdown(), "Hello **MoonBit**")
  assert_eq(doc.query("p.intro").length(), 1)
  let tokens = @html_parser.tokenize("<p>Hello</p>").tokens
  assert_eq(tokens.length(), 4)
}

///|
test "readme parse bytes example" {
  let bytes = @utf8.encode("<meta charset=utf-8><p>\u{20AC}</p>")
  let doc = @html_parser.parse_bytes(bytes)
  guard doc.encoding is Some("utf-8") else { fail("expected utf-8 encoding") }
  assert_eq(doc.to_text(separator="", strip=false), "\u{20AC}")
}

///|
test "readme sanitize dom example" {
  let doc = @html_parser.parse(
    "<!DOCTYPE html><!--x--><p onclick=alert(1)>ok</p><script>alert(1)</script>",
    sanitize=false,
  )
  let clean = @html_parser.sanitize_dom(
    doc.root,
    policy=@html_parser.default_sanitization_policy(),
  )
  assert_eq(@html_parser.to_html(clean, pretty=false), "<p>ok</p>")
  let fragment = @html_parser.parse_fragment(
    "<p onclick=alert(1)>ok</p><script>alert(1)</script>",
    sanitize=true,
  )
  assert_eq(fragment.to_html(pretty=false), "<p>ok</p>")
}

///|
test "readme CLI reader example" {
  let paths : Array[String] = []
  let result = @html_parser.run_cli_with_reader(["-", "--format", "text"], fn(
    path,
  ) {
    paths.push(path)
    @utf8.encode("<p>Hello <b>MoonBit</b></p>")
  })
  @test.assert_eq(paths, ["-"])
  assert_eq(result.exit_code, 0)
  assert_eq(result.stdout, "Hello MoonBit\n")
}

#Native CLI

The native CLI wrapper lives in cmd/main and uses moonbitlang/async for raw stdin/file IO, stdout/stderr, and output files without custom C stubs. Build it from this repository with:

moon run --target native --release --build-only cmd/main

The executable is written to _build/native/release/build/bobzhang/html_parser/cmd/main/main.exe. For example:

printf '<p>Hello <b>MoonBit</b></p>' \
  | _build/native/release/build/bobzhang/html_parser/cmd/main/main.exe - --format text

Black-box CLI integration tests live in tests/scrut and run with:

moon run --target native scripts/check_scrut_cli.mbtx

#Workspace Examples

This repository also has a moon.work workspace with an examples module for runnable documentation. The formatter example uses the local parser checkout:

moon run --target native examples/cmd/htmlfmt -- "<article><p>Hello <b>MoonBit</b></p></article>"

The examples documentation lives in examples/README.mbt.md.

#Development Checks

Run the same validation entrypoint used by CI:

moon run --target native scripts/check_ci.mbtx --skip-without-credentials

Drop --skip-without-credentials when logged in locally and checking the full Mooncakes dry-run path. The script checks release-version consistency, MoonBit script inventory and argument smoke paths, validation-inventory wiring, GitHub workflow drift and tracked workflow inventory including the Copilot setup workflow, source layout, test-name inventory, migration docs, local Git hook wiring, vendored fixture sync when .repos/justhtml is present, vendored fixture manifest hashes, package metadata, formatting, generated interfaces, all supported targets, default/JS/native tests with a count floor, coverage, native CLI smoke behavior, Scrut CLI integration tests, Mooncakes package validation, and dynamic Mooncakes archive inventory/content checks.

#
CliReadPlan

source

Describes how the CLI should obtain input after argument parsing.

CliImmediate means parsing already produced a complete result, such as help, version output, or an argument error. CliReadPath asks the caller to read the given path, where "-" conventionally means standard input.

#
CliResult

source

Result produced by the embeddable CLI runner.

stdout and stderr contain terminal output. When file_output_path is present, file_output_content contains the bytes that a shell wrapper should write to that path instead of printing to standard output.

#
DecideAction

source

Action returned by a TransformSpec::decide callback.

Keep leaves the matched node unchanged. The other variants apply the same structural operations as the corresponding selector transforms.

#
DisallowedTagHandling

source

How sanitizer handles elements whose tag names are not allowlisted.

#
DoctypeInfo

source

Doctype data emitted by the tokenizer.

force_quirks is set when the tokenizer sees a malformed or legacy doctype form that should place a parsed document into quirks mode.

#
FragmentContext

source

Context element used when parsing an HTML fragment.

The tag name is normalized to lowercase. ns can be used for foreign content contexts such as SVG or MathML.

#
HtmlContext

source

Output context used by HTML serialization.

Non-Html contexts escape serialized output for embedding in JavaScript strings, HTML attribute values, or URL text.

#
HtmlError

source

Error type raised by strict parsing, serialization, sanitizer, and selector operations.

#
HtmlToken

source

Token variants produced by the HTML tokenizer.

Eof is appended as the final token. Character data and comments are stored as already-normalized MoonBit strings; parse diagnostics are returned through TokenizedHtml.errors when error collection is enabled.

#
LinkMatch

source

A URL or email-like span found in plain text.

start and end are UTF-16 offsets into the input StringView. kind is "url" or "email".

#
LinkifyConfig

source

Options for the plain-text link scanner.

#
Node

source

DOM node used for documents, fragments, elements, text, comments, and doctypes.

Nodes are created with helpers such as document, fragment, element, text, comment, and doctype.

#
NodeKind

source

Kind of DOM node represented by Node.

#
ParseError

source

Parser or tokenizer diagnostic with optional source location.

category identifies the source of the error, such as tokenizer or treebuilder. message defaults to code when no custom message is given.

#
ParsedHtml

source

Result of parsing HTML.

root is the document or fragment root. errors is populated when collect_errors=true or strict parsing observes an error. encoding is set by byte parsing APIs.

#
SanitizationPolicy

source

DOM sanitization policy.

A policy controls allowed tags and attributes, URL filtering, comment and doctype handling, foreign-content hardening, CSS style allowlists, selector limits used by transform hooks, and unsafe-input reporting.

#
SanitizeTransformObserver

source

Observer callbacks for sanitizer-driven DOM rewrites.

The node hook runs for events that have an associated DOM node. The report callback receives every sanitizer event message and the optional related node, including unsafe input that was stripped or collected.

#
SelectorLimits

source

Resource limits used while parsing and matching CSS selectors.

Limits are defensive bounds for selector length, nesting, list size, and match cost. Negative match budgets are treated as exhausted.

#
StreamDoctypeEvent

source

Doctype event emitted by the streaming tokenizer facade.

#
StreamEvent

source

Token-level streaming event.

The stream API does not build a DOM tree. It forwards start tags, end tags, text, comments, and doctypes in tokenizer order, coalescing adjacent text.

#
StreamSink

source

Mutable sink that converts tokenizer tokens into coalesced stream events.

#
StreamStartEvent

source

Start-tag event emitted by the streaming tokenizer facade.

name is the normalized tag name and attrs contains the decoded attributes for the tag.

#
TagKind

source

Whether a tag token opens an element or closes one.

StartTag represents tags such as <p> and EndTag represents tags such as </p>.

#
TagToken

source

A start or end tag emitted by tokenize.

name is lower-cased for HTML tag tokens. attrs maps attribute names to optional values; a value of None represents a minimized or missing-value attribute. self_closing records the solidus marker on start tags.

#
TokenizedHtml

source

Result returned by tokenize.

tokens always ends with HtmlToken::Eof. errors is empty unless collect_errors=true was passed.

#
TransformSpec

source

A DOM transform specification for apply_transforms.

The current port covers deterministic structural, attribute, callback, URL/style, utility, and sanitizer transforms. Most selector and node-kind transforms also support hook/report callbacks.

#
UnsafeHandling

source

How sanitizer reports unsafe input that it strips or rewrites.

#
UrlFilter

source

Callback wrapper used to rewrite or reject URL values before validation.

The callback receives normalized tag name, normalized attribute name, and the raw attribute value. Returning None drops the URL.

#
UrlHandling

source

Action used after a URL value passes the configured URL checks.

#
UrlPolicy

source

URL sanitization policy shared by URL-bearing attributes.

Exact (tag, attr) rules take precedence. Unmatched URL-like attributes use the default handling and relative-URL behavior.

#
UrlPolicyRule

source

URL rule bound to a tag and attribute name.

#
UrlProxy

source

Proxy endpoint used when a URL rule selects UrlProxy.

Sanitized URLs are emitted as url?param=<encoded-url> or url&param=<encoded-url> depending on whether the proxy URL already has a query string.

#
UrlRule

source

Per-attribute URL validation rule.

Rules can restrict schemes and hosts, disallow fragments, normalize protocol-relative URLs, override relative-URL handling, or route accepted URLs through a proxy.

#
apply_transforms

source

Apply transforms to a DOM tree in order.

The transform mutates and returns node.

#
cli_help

source
fn cli_help() -> String

Return the CLI usage text.

#
cli_read_plan

source

Parse CLI arguments and report whether the caller must read an input path.

#
comment

source
fn comment(data : StringView) -> 
Node

Create a comment node.

#
css_preset_text

source
fn css_preset_text() -> Array[String]

Return the conservative CSS property allowlist for text styling.

#
default_document_sanitization_policy

source
fn default_document_sanitization_policy() -> 
SanitizationPolicy

Return the default document sanitization policy.

#
default_sanitization_policy

source

Return the default fragment sanitization policy.

#
doctype

source
fn doctype(name? : String, public_id? : String, system_id? : String, force_quirks? : Bool) -> 
Node

Create a doctype node.

#
document

source

Create a document node with optional children.

#
element

source
fn element(name : StringView, attrs? : Map[String, String?], children? : Array[
Node], ns? : String) -> 
Node

Create an element node.

Namespace aliases html, svg, and mathml are normalized for serializer and sanitizer behavior. Child nodes are attached in order.
fn find_links(text : StringView) -> Array[
LinkMatch]

Find URL and email-like spans in plain text using the default configuration.

Find URL and email-like spans in plain text using an explicit configuration.

#
fragment

source

Create a document-fragment node with optional children.

#
linkify_dom

source

Linkify URL and email text inside a DOM subtree.

The transform mutates and returns node.

#
matches

source
fn matches(node : 
Node, selector : StringView) -> Bool

Return whether node itself matches a CSS selector.

#
parse

source
fn parse(html : StringView, sanitize? : Bool, collect_errors? : Bool, strict? : Bool, scripting_enabled? : Bool, xml_coercion? : Bool, track_node_locations? : Bool) -> 
ParsedHtml raise 
HtmlError

Parse a full HTML document.

#
parse_bytes

source
fn parse_bytes(input : BytesView, encoding? : String, sanitize? : Bool, collect_errors? : Bool, strict? : Bool, scripting_enabled? : Bool, xml_coercion? : Bool, track_node_locations? : Bool) -> 
ParsedHtml raise 
HtmlError

Decode bytes and parse a full HTML document.

#
parse_fragment

source
fn parse_fragment(html : StringView, context? : 
FragmentContext, sanitize? : Bool, collect_errors? : Bool, strict? : Bool, scripting_enabled? : Bool, xml_coercion? : Bool, track_node_locations? : Bool) -> 
ParsedHtml raise 
HtmlError

Parse an HTML fragment.

#
query

source

Return all descendants of root that match a CSS selector.

#
query_one

source
fn query_one(root : 
Node, selector : StringView) -> 
Node?

Return the first descendant of root that matches a CSS selector.

#
run_cli_bytes

source
fn run_cli_bytes(args : ArrayView[String], input : BytesView) -> 
CliResult

Run the embeddable CLI with already-read input bytes.

#
run_cli_with_reader

source
fn run_cli_with_reader(args : ArrayView[String], read_input : (String) -> Bytes) -> 
CliResult

Run the embeddable CLI with a path reader callback.

#
sanitize_dom

source

Sanitize a DOM node in place and return the sanitized root.

When no policy is supplied, document roots use the document policy and other roots use the fragment policy.

#
stream

source

Parse an HTML string into streaming events.

#
stream_bytes

source
fn stream_bytes(input : BytesView, encoding? : String) -> Array[
StreamEvent]

Decode HTML bytes and return streaming events.

When encoding is omitted, the same byte-sniffing path used by parse_bytes chooses the input encoding.

#
stream_bytes_each

source
fn stream_bytes_each(input : BytesView, emit : (
StreamEvent) -> Unit, encoding? : String) -> Unit

Decode HTML bytes and emit streaming events incrementally.

When encoding is omitted, the same byte-sniffing path used by parse_bytes chooses the input encoding.

#
stream_each

source
fn stream_each(html : StringView, emit : (
StreamEvent) -> Unit) -> Unit

Parse an HTML string and emit streaming events incrementally.

#
text

source
fn text(data : StringView) -> 
Node

Create a text node.

#
to_html

source
fn to_html(node : 
Node, pretty? : Bool, indent_size? : Int, context? : 
HtmlContext, quote? : Char) -> String raise 
HtmlError

Serialize a node as HTML.

#
to_markdown

source
fn to_markdown(node : 
Node, html_passthrough? : Bool) -> String raise 
HtmlError

Render a DOM node and its descendants as Markdown.

#
to_test_format

source
fn to_test_format(node : 
Node) -> String

Render a deterministic tree dump intended for conformance tests.

#
to_text

source
fn to_text(node : 
Node, separator? : String, strip? : Bool, separator_blocks_only? : Bool) -> String

Extract descendant text from a node.

#
tokenize

source
fn tokenize(html : StringView, collect_errors? : Bool, xml_coercion? : Bool) -> 
TokenizedHtml

Tokenize an HTML string without building a DOM tree.

Set collect_errors=true to collect tokenizer diagnostics in the returned TokenizedHtml. Set xml_coercion=true to replace XML-invalid text and comment characters during tokenization.

Source Files

Powered by MoonBit

Site sourceReport issuePackagesBuild queueSkillsStatistics

© 2026 mooncakes.io