textkernel

    A formally verified MoonBit kernel for HTML escaping, filename sanitisation, URL rewriting and joined-list batch operations.

    html
    escaping
    sanitisation
    formal-verification
    string
    Download zip
    Version
    0.1.4
    License
    MIT
    Last updated
    3 hours ago
    Downloads
    16

    No dependencies

    #textkernel

    quality proof wasm compat mooncakes skills license

    A small, formally verified MoonBit kernel for text that crosses a trust boundary.

    When a program takes a string from one system and puts it into another — into HTML, into a filename, into a URL, into a batch API — four things can go wrong: markup can be injected, a filename can escape its directory, a URL can be rewritten incorrectly, and a batch loop can drop an item. textkernel is the set of routines for those four jobs, with the safety properties discharged by moon prove rather than by inspection.

    huanguan-Q/textkernel 151 tests × 4 backends 419 proof goals 13 WASM API exports (+ `_start`) zero deps

    #What is in the box

    RoutineWhat it guarantees
    escape_html / escape_attrOutput contains no raw < > (nor " ' for attributes); every & starts a complete entity the library emits
    sanitize_filenameReplaces the nine unsafe characters and lone surrogates, clamps to 120 code units, never returns empty
    normalize_asset_urlForces https for configured hosts and appends missing default query params; the tail is carried over verbatim
    decide_referrer_policyPicks a Referrer-Policy from a configured host allow-list
    str_containsUTF-16 code-unit substring test (the host gate above)
    split_joined / join_joinedExact inverse pair for batch calls; slices by code unit so non-BMP characters survive
    batch_process_customApplies an operation in bounded chunks; a non-positive chunk size is clamped to 1, never divided by

    *_result variants return Result[T, CoreError] with an explicit input bound. The shipping layer also checks its own output against the invariants before returning and aborts on violation — a regression fails loudly instead of quietly emitting markup the caller believes is escaped.

    #Reproduce

    moon fmt --check moon check --target all --deny-warn # wasm / wasm-gc / js / native, zero warnings moon test --target all # 151/151 on each backend moon prove # 191 goals with the stock solver limits node scripts/prove_gate.mjs # 419 goals, 10x solver limits in CI, machine-int model moon run src/demo --target native # runnable example moon package --list # validate package contents without releasing node scripts/build_wasm.mjs # build wasm/textkernel.wasm node examples/wasm-node/demo.mjs # call the WASM exports from JavaScript node examples/wasm-node/render.mjs # downstream consumer example

    Two proof counts appear above because they answer different questions: the stock moon prove run discharges 191 goals, while scripts/prove_gate.mjs retries at higher solver limits (5x by default, 10x in CI) under the 32-bit machine-integer prelude and discharges 419. The gate is what CI uses; the extra budget is needed because the newest CLI leaves one goal timing out at 5x.

    Toolchain used to produce the numbers above: moon 0.1.20261104, Why3 1.7.2, z3. The proof verdicts depend on the VC generator, so pin the toolchain when reproducing; the historical pin is kept here for provenance.

    Re-verified 2026-11-19 on the local toolchain moon 0.1.20261117 (eb64713): 146/146 tests pass on all four backends, moon prove discharges 191 goals, node scripts/prove_gate.mjs discharges 419, and moon check --target all--deny-warn is clean after migrating StringBuilder(size_hint=...) to the official StringBuilder(capacity=...) spelling (warning 0020).

    Release update 2026-11-22: the dated CLI binaries for 0.1.20261104 and 0.1.20261117 are no longer downloadable, so CI now installs the latest available MoonBit CLI and runs the proof job on ubuntu-22.04 (Ubuntu 26.04's OCaml 5.4 conflicts with why3 1.7.2). Re-checked locally on moon 0.1.20261119: format, check and all 151x4 tests are green; scripts/prove_gate.mjs at the default 5x budget reports 425 goals proved with 1 timeout and fails, while PROVE_TIME_SCALE=10 proves all 419 goals. CI therefore uses the 10x budget. Treat the proof result as environment-sensitive and do not bump the prover without re-establishing the gate.

    #Use

    let safe = @lib.escape_html("<script>alert(1)</script>")
    // &lt;script&gt;alert(1)&lt;/script&gt;

    let name = @lib.sanitize_filename("report/2026:Q4?.md")
    // report_2026_Q4_.md

    let items = @lib.split_joined("a|b|c", "|")
    let out = @lib.join_joined(@lib.quick_batch_escape_htmls(items), "|")

    See src/demo/main.mbt for a complete runnable example and examples/wasm-node/ssr-middleware.mjs for an SSR/UGC consumer.

    #Explicit configuration and CLI

    Host allow-lists and default query parameters are data, not source constants:

    let config = @lib.create_asset_config(
    ["cdn.internal.example"],
    ["w=800", "auto=1"],
    )
    let url = @lib.normalize_asset_url_with_config(
    "http://cdn.internal.example/a.png",
    config,
    )
    // https://cdn.internal.example/a.png?w=800&auto=1

    The no-suffix functions remain wrappers around the historical RFC 2606 configuration, so existing callers do not break.

    The same operations are available as a CLI with JSON-lines output:

    moonx huanguan-Q/textkernel/src/main@0.1.4 -- --help moonx huanguan-Q/textkernel/src/main@0.1.4 -- escape-html '<script>x</script>' moon run src/cli --target native -- sanitize-filename < names.txt

    Use the explicit src/main package path for moonx: the bare module coordinate has no prebuilt Wasm asset on mooncakes.io.

    #WASM / calling from JavaScript

    src/main is an export shim plus a small CLI entry point. With use-js-builtin-string the linker exposes String -> String functions directly to a JS host — no linear-memory ABI, no manual allocation, no UTF-8 copy:

    Single valueJoined-list batch (joined, sep)
    escape_html, escape_attr, sanitize_filenameescape_htmls_joined, escape_attrs_joined, sanitize_filenames_joined
    normalize_asset_url, decide_referrer_policy, str_containsnormalize_asset_urls_joined
    escape_html_safe, escape_attr_safe, sanitize_filename_safe— never-fail variants

    node scripts/build_wasm.mjs # ~23 KB artefact after the CLI entry point; verifies 13 API exports (+ `_start`) node examples/wasm-node/demo.mjs # assertions across all 13 API exports node examples/wasm-node/render.mjs # realistic downstream consumer

    examples/wasm-node/loader.mjs reads the module's import table and supplies the imported string constants as plain JS globals. That works on Node 22+ and Chrome alike: Chrome can synthesise them with the importedStringConstants compile option, but Node 22 does not implement that option, so the loader does not depend on it. The builtin wasm:js-string functions come from new WebAssembly.Module(bytes, { builtins: ["js-string"] }).

    The committed wasm/textkernel.wasm is a convenience copy so the example runs straight after a clone. CI rebuilds from source on every push and runs the example again, so a stale artefact cannot claim to be working code.

    #Error handling

    The production path never aborts. The *_result variants return Result[String, CoreError] and cover both failure sources: an input outside the shipping domain (InputTooLarge) and a violated output invariant (InvariantViolation). CoreError carries both alongside the original InvalidInput / PerformanceWarning; error_to_string renders all four.

    The String-returning escape_html / escape_attr / sanitize_filename are a compatibility layer kept for existing callers and the WASM ABI. They route every failure through apply_failure_policy — the single place in the library that can abort. Callers choose a policy explicitly with *_with_policy:

    PolicyBehaviourUse
    FailFast (compatibility default)abortTrusted inputs; preserves historical semantics
    EmptyOnErrorreturn ""Trust boundaries where a trap would take down the host

    "" is safe for every kernel: it is neither markup nor an unsanitised filename. *_safe variants and their three WASM exports use EmptyOnError.

    examples/wasm-node/render.mjs exercises the whole thing as a downstream consumer would: it renders untrusted comments into HTML, escapes text nodes and attributes separately, rejects an 8,000,001-code-unit payload without trapping the host, and shows that the fail-fast export still traps — now an explicit policy choice instead of the only behaviour available.

    #Boundaries — read this before relying on it

    • The prover has no model for String. The proved kernels in src/proof operate on FixedArray[Int32] UTF-16 code units. src/proof/conform.mbt pins each kernel to the shipped String implementation over an enumerated corpus. That two-line conversion is the trusted seam; it is a corpus equivalence, not a theorem about String.
    • Only src/proof is proof-enabled. src/lib and src/demo are not in the moon prove denominator — proof_enabled is a whole-package switch and the String/Array usage in src/lib is outside the current model. The runtime invariant checks in src/lib/invariants.mbt are what close that gap.
    • Input bounds are arithmetic, not resource limits. MAX_INPUT_CODE_UNITS keeps the kernels' 5 * length / 6 * length inside 32 bits; MAX_SHIPPING_CODE_UNITS (8,000,000 code units) is the tighter bound the shipping layer enforces. Both are documented in invariants.mbt.
    • batch.mbt proves the arithmetic, not scheduling, concurrency or timing. BatchConfig now has exactly one field (batch_size): the earlier max_concurrent / enable_memory_pool / timeout_ms were removed because a pure function has no scheduler, pool or clock to honour them.
    • The WASM ABI requires JS string builtins. It is not usable from a host without wasm:js-string support (e.g. a plain wasmtime embedder). A linear-memory ABI is not currently provided.
    • The host allow-list and default query params are data, passed through AssetConfig. The no-suffix functions keep the historical RFC 2606 example domains (cdn.example.com, assets.example.org) as a compatibility default.
    • No verified third-party adoption yet. There are no downstream dependents outside this repository's own examples. ssr-middleware.mjs shows the shape a consumer would write; it is not evidence of adoption.

    #Performance

    The escapers and the filename sanitiser used to build their result with out = out + fragment inside a loop. That copies the whole prefix on every iteration, which made the native backend quadratic: 4x input produced ~8x runtime. They now write into a StringBuilder, which is amortised O(n).

    Measured on moon 0.1.20261104, native release, 100 rounds of escape_html (identical harness before and after, inclusive of process startup):

    Input (UTF-16 code units)BeforeAfterSpeedup
    20,4806.06 s0.49 s12x
    81,92050.54 s0.56 s90x
    327,680187.8 s / 20 rounds0.73 s / 100 rounds~1300x per round

    A 1000-round run after the fix (which also reads the produced string's tail, so the work cannot be optimised away) stays linear: 0.63 s / 0.90 s / 2.35 s for 20k / 80k / 327k code units. The JS backend was already linear and is unchanged: 0.12 s / 0.34 s for 20k / 80k at 100 rounds.

    Output is byte-for-byte identical. src/lib/string_builder_regression_test.mbt keeps the original implementations as private oracles and asserts equality over an enumerated corpus plus 200 deterministic pseudo-random strings, on all four backends. The proof kernels and their 419 goals are untouched — they operate on code-unit arrays, and src/proof/conform.mbt still pins them to the shipped String API.

    To reproduce, export the tree with git archive HEAD, build a small release executable that calls escape_html N times over a string grown by doubling "a<&>x", and time it with /usr/bin/time -p. Subtract a baseline variant that only measures input.length() to remove process startup, and use the identical harness before and after a change. The numbers are a regression guard, not a promise: absolute throughput depends on the backend and host.

    #Evidence and further reading

    • docs/BENCHMARK.md — recorded native timings and reproduction method.
    • docs/PROOF-REPORT.md — what the proof gates establish and where the trusted seam is.
    • docs/FORMAL-VERIFICATION-ARTICLE.md — a reproducible walkthrough of the proof design.
    • docs/ABI-EVALUATION.md — why the current Wasm artefact needs JS string builtins and the proposed WASI path.
    • docs/TOOLCHAIN-PIN.md — why the CI compatibility pin is a repository variable.
    • Mooncakes Skills entry: https://mooncakes.io/skills/huanguan-Q/textkernel@0.1.4.
    • SKILL.md — agent-facing command reference.
    • docs/API-STABILITY.md — SemVer contract and stable API surface.
    • CHANGELOG.md — release history.
    • CONTRIBUTING.md — development and review gates.
    • SECURITY.md — supported versions and private reporting.

    #Versioning

    This project follows Semantic Versioning 2.0.0. Patch releases are output-preserving bug fixes; minor releases add APIs with compatibility defaults; major releases may remove or change stable names. The stable surface and deprecation path are listed in docs/API-STABILITY.md.

    #Provenance & License

    All code here is an independent implementation. It was extracted from a larger private codebase by the same author and generalised for this repository: platform-specific host names and query parameters were replaced with reserved example domains, and two layers were rewritten from scratch to remove scaffolding dependencies —

    • src/lib/validation.mbt: rewritten without the original logging side effects.
    • src/lib/batching.mbt: rewritten without the original content-analysis and wall-clock modules; the arithmetic it uses is the exact expression discharged by src/proof/batch.mbt.

    No third-party source code is included. No code was translated or ported from another project. The proof obligations, specifications and test corpora are original work.

    Licensed under MIT — see LICENSE.