toon4

    TOON v4.4 codec, line/column diagnostic engine, and JSON vs TOON token-efficiency advisor in MoonBit

    toon
    serialization
    llm
    parser
    diagnostics
    Download zip
    Author
    Version
    0.1.0
    License
    MIT
    Last updated
    13 hours ago
    Downloads
    6

    No dependencies

    #toon4 — TOON v4.4 Codec, Pinpoint Diagnostics & Token-Efficiency Advisor

    Pure MoonBit implementation of TOON (Token-Oriented Object Notation) v4.4, built to:

    1. Close TOON v4.4 compatibility gaps in existing MoonBit tooling (including keyed tabular form users[2:]{name,score}:, nested field groups geo{lat,lon}, comment pre-pass #, canonical [] empty arrays, strict UTF-8 & number parsing, and exact §14.4 non-strict recoveries) — 564 / 564 official toon-format/spec fixtures passing.
    2. Pinpoint errors with actionable diagnostics — structured DecodeError values carry an ErrorKind code, 1-based line/column Span, tab visualization (→), and ^ caret underline pointing directly at the offending token.
    3. Measure whether converting to TOON actually pays off — compares Pretty JSON, Compact JSON, and TOON (comma, tab, pipe), reports exact UTF-8 byte sizes, tabular eligibility coverage, path-level fallback reasons, and an explicit RECOMMEND_TOON / MARGINAL / KEEP_COMPACT_JSON verdict.


    #Why toon4? (Compatibility Gap vs oliveagle/toon-mbit)

    Auditing oliveagle/toon-mbit against TOON Spec v4.4 and MoonBit 0.10.21 (moon 0.1.20261117) identified the following verified gaps:

    Capability / Spec Clauseoliveagle/toon-mbit (audited)Waysyou/toon4
    Toolchain manifest (moon.mod / moon.pkg)Uses legacy moon.mod + moon.pkg.json (rejected by moon 0.1.20261117)Modern moon.mod + moon.pkg DSL; zero warnings (--deny-warn) across wasm, wasm-gc, js, native
    TOON 4.4 §9.5 Keyed Tabular (users[2:]{name,score}:)Fails at src/decoder.mbt:273-295, 360-369 (: inside [2:] treated as key-value colon / illegal symbol)Full encode, decode, nested field groups, [0:], delimiter support, and strict/non-strict rules
    §9.3 Nested Field Groups (regions[2:]{name,geo{lat,lon}}:)Not supportedRecursive FieldNode ADT (Leaf / Group) in both encoder and decoder
    §11 Tab Delimiter Header (items[2\t]:)src/decoder.mbt:299 .trim() strips trailing \t before parsing bracket delimiterPreserves exact HTAB and | header delimiters; validates {fields} delimiter consistency
    §5.1 Full-line Comments (# ...)Not strippedPre-pass strips space-indented # lines before blank-line, quote, and indent checks
    §9.1 / §9.2 Empty ArraysEncodes deprecated [0]: / key[0]: (src/encoder.mbt:194-209); decodes [] as stringEmits [] / key: [] / - [0]:; decodes [] as array in root/field/list-item positions and "[]" inside cells
    §4 Strict Numbers (1e5, 05, +1, 1.)05 parsed as integer 5; 1e5 fails parse_doubleExact §4 DFA (1e5 -> 100000, 05 / +1 / 1. / 0x10 stay strings, -0.0 -> 0)
    §7.1 Escapes & Quotes (\uXXXX, "a" "b")Rejects \uXXXX; unquote_string skips interior quote check ("a" "b" passes)Full \uXXXX + surrogate rejection; strict closing-quote boundary check
    §14 Strict vs 5 Non-Strict RecoveriesChecks strict only on inline array counts (src/decoder.mbt:401)Full strict enforcement + all 5 §14.4 non-strict recoveries (preserving key insertion order via Map)
    UTF-16 Emoji Index SafetyMixes Char iterator offsets with UTF-16 slices (src/decoder.mbt:593-612)Scans UTF-16 code units consistently; maps offsets to 1-based Unicode scalar columns for carets
    Error DiagnosticsParseError { line, message } hardcodes line: 0 in all 6 header errorsDecodeError with ErrorKind, Span(line, column, length), source line, and caret formatting


    #Official Conformance & Verification

    Vendored all 23 official fixture files from toon-format/spec (v4.4.0, MIT License) under conformance/fixtures/:

    • Encode fixtures: 160 / 160 passing (9 files)
    • Decode fixtures: 404 / 404 passing (14 files, including 158 shouldError: true cases, strict object key insertion-order check, and valid span.line >= 1 check)
    • Total official fixtures: 564 / 564 (100%) passing on both native and wasm
    • Additional suites: Raw ill-formed UTF-8 byte sequence rejection, TOON 4.4 differential tests, caret golden snapshots, CLI integration tests, and advisor decision tests — all running with zero warnings under moon check --deny-warn.


    #Pinpoint Diagnostics Example

    DecodeError[E_WIDTH] at line 3, column 10: expected 3 keyed entry cells, but got 2 3 | WH-SZ: Shenzhen,1 | ^^^^^^^^^^

    DecodeError[E_INDENT] at line 2, column 1: tabs are not allowed in indentation in strict mode 2 | →b: 1 | ^


    #Cost & Token-Efficiency Advisor (Real Output)

    Running moon run examples/inventory_agent --target native on an 80-SKU inventory list + 3 keyed warehouses (warehouses[3:]{city,leadDays,active}:):

    TOON v4.4 Cost & Token-Efficiency Analysis ------------------------------------------------------------------------ JSON (pretty) ████████████████████ ~4835 tok (14113 B, 7388 syntax B, +79.3% vs compact) JSON (compact) ███████████░░░░░░░░░ ~2696 tok (9353 B, 2628 syntax B, 0.0% vs compact) * TOON (comma) ███████░░░░░░░░░░░░░ ~1584 tok (4139 B, 686 syntax B, -41.2% vs compact) TOON (tab) ███████░░░░░░░░░░░░░ ~1587 tok (4141 B, 688 syntax B, -41.1% vs compact) TOON (pipe) ███████░░░░░░░░░░░░░ ~1587 tok (4141 B, 688 syntax B, -41.1% vs compact) ------------------------------------------------------------------------ Tabular coverage: 100.0% (tabular_items=83, fallback_items=0, array_blocks=1, keyed_blocks=1) Verdict [RECOMMEND_TOON]: Recommend TOON (comma): -41.2% estimated tokens vs compact JSON (-67.2% vs pretty JSON), tabular coverage 100.0%.

    Conversely, running moon run --target native cmd/toon4 -- analyze conformance/fixtures/encode/objects-keyed.json on semi-uniform nested fixture data accurately warns against converting:

    Tabular coverage: 48.6% (tabular_items=18, fallback_items=19, array_blocks=0, keyed_blocks=9) Fallback paths: - tests (14 items): element [1] adds extra key 'note' absent in [0]; falls back to list form - tests[7].input.config (1 items): object has only 1 entry (< 2 minimum for TOON 4.4 keyed tabular '[N:]') - tests[11].input.items (2 items): element [1] is missing key 'config' present in [0]; falls back to list form - tests[12].input.items (2 items): contains non-object, empty-object, or nested-array elements; uses list form ('- ') Verdict [KEEP_COMPACT_JSON]: Keep compact JSON: TOON (comma) adds +19.1% estimated tokens vs compact JSON (tabular coverage 48.6%; 4 non-tabular path(s)).


    #Performance Benchmarks (moon bench)

    Measured directly with moon bench -p Waysyou/toon4/benchmarks (moonc v0.10.21+06da1ed44):

    Benchmarknative (mean ± σ)wasm / moonrun (mean ± σ)
    encode_inventory_200 (200 SKU rows, 8 cols)411.67 µs ± 10.37 µs479.54 µs ± 16.71 µs
    decode_inventory_200 (200 SKU rows, 8 cols)1.07 ms ± 78.63 µs1.09 ms ± 28.40 µs
    encode_keyed_flags_80 (80 keyed entries, [80:])145.00 µs ± 2.32 µs144.37 µs ± 2.86 µs
    decode_keyed_flags_80 (80 keyed entries, [80:])274.56 µs ± 10.65 µs281.96 µs ± 6.96 µs
    analyze_inventory_200 (3 TOON + 2 JSON + audit)2.94 ms ± 129.90 µs4.25 ms ± 295.95 µs
    analyze_nested_orders_50 (50 nested orders)1.59 ms ± 42.72 µs2.30 ms ± 80.30 µs


    #Architecture & Scope Boundaries

    #Package Layout

    • types: Core ADTs (Delimiter, FieldNode, ArrayHeader, StreamEvent), options, Span, ErrorKind, DecodeError::format, EncodeError.
    • lex: UTF-16-safe lexical scanner, §1.2/§12 whitespace & comment stripping, §4 number DFA, §7 escaping/quoting, §6/§9.5 header parser.
    • encode: Canonical number normalization (§2/§3), recursive FieldNode tabular & keyed-tabular eligibility, streaming encode_to_lines.
    • decode: Pull-based lookahead LineReader, streaming decode_stream emitting StreamEvent, EventBuilder, strict UTF-8 decode_bytes.
    • advisor: BPE-aware offline token estimator, recursive structural eligibility auditor, multi-format analyze & Markdown/JSON reporting.
    • toon4 (root): Unified public API facade re-exporting core & advisor types and entrypoints.
    • cli + cmd/toon4: Testable CLI command logic (cli) + async file/stdio executable (cmd/toon4).
    • conformance, benchmarks, examples/inventory_agent: Official 564-case spec runner, differential diagnostics, @bench.T suite, and warehouse agent walkthrough.

    #Explicit Scope Boundaries & Design Trade-offs

    Numeric precision (§2 and §4). The CLI preserves source number text through JSON → TOON → JSON and normalizes its decimal spelling without rounding digits. The decoder stores that text in Json::Number.repr; reading just its host Double value can still round or underflow. JSON or TOON literals that overflow the supported finite Double range fail with a diagnostic. For library JSON input, parse with preserve_number_repr=true to retain the original digits.

    1. Offline BPE-Aware Token Estimator: advisor uses a deterministic, zero-dependency BPE-style heuristic (modeling o200k/cl100k 3-digit chunking, identifier splitting, JSON punctuation clusters, and newline+indent merging) rather than embedding multi-megabyte external .tiktoken BPE tables. UTF-8 byte counts, syntax byte counts, and tabular coverage percentages are exact; token counts are labeled estimated_tokens and may vary slightly across specific model tokenizers.
    2. TOON v4.0+ Literal Dotted Keys: Per toon-format/spec v4.0–v4.4, keyFolding / expandPaths are intentionally absent; dotted keys like a.b are unconditionally treated as literal keys.
    3. Target Compatibility: All core packages (types, lex, encode, decode, advisor, cli, root toon4, examples/inventory_agent, benchmarks) compile and pass on all 4 backends (wasm, wasm-gc, js, native). File-system I/O packages (cmd/toon4 and conformance/spec_conformance_test.mbt) use moonbitlang/core/async/fs and target native + wasm.
    4. Small-Payload Prompt Tax: Even when TOON reduces raw payload characters on tiny 2–3 row snippets, advisor marks payloads < 45 compact tokens (or < 12 tokens saved) as MARGINAL because LLM format instructions carry a fixed prompt overhead.


    #Quick Start

    #Library Usage (moon.pkg)

    import { "Waysyou/toon4", }

    let doc = Json::object(Map([
    ("users", Json::object(Map([
    ("alice", Json::object(Map([("name", Json::string("Alice")), ("score", Json::number(95.0))]))),
    ("bob", Json::object(Map([("name", Json::string("Bob")), ("score", Json::number(88.0))]))),
    ]))),
    ]))

    let toon_text = @toon4.encode(doc)
    let round_trip = @toon4.decode(toon_text)
    let report = @toon4.analyze(doc)
    println(report.render_report())

    #CLI Usage

    # Compare JSON vs TOON efficiency and inspect non-tabular fallback paths moon run --target native cmd/toon4 -- analyze data.json moon run --target native cmd/toon4 -- analyze --json data.json # Encode JSON to TOON v4.4 (comma | tab | pipe) with stats to stderr moon run --target native cmd/toon4 -- encode -d pipe --stats data.json # Validate TOON with pinpoint line/column caret diagnostics moon run --target native cmd/toon4 -- check payload.toon # Decode TOON (strict by default, or --no-strict for §14.4 recoveries) moon run --target native cmd/toon4 -- decode payload.toon

    #Running All Tests & Benchmarks

    moon check --deny-warn --target all moon test --target native moon test --target wasm moon test --target wasm-gc moon test --target js moon bench -p Waysyou/toon4/benchmarks --target native


    #License

    MIT — see LICENSE. Official conformance fixtures in conformance/fixtures/ are MIT-licensed from toon-format/spec.

    ArrayHeader

    Parsed array or keyed-tabular header metadata (TOON §6 & §9.5).

    CostReport

    Complete cost-efficiency and tabular diagnostic report for a JSON document.

    DecodeError

    Structured TOON decode error carrying diagnostic kind, human message, 1-based line/column span, and optional offending source line.

    DecodeOptions

    Options controlling TOON decoding (TOON §13.2).

    Delimiter

    Delimiter character used in array headers, inline arrays, tabular rows, and keyed tabular entry rows per TOON §11.

    EncodeError

    Structured TOON encode error (e.g. invalid options or unpaired UTF-16 surrogates).

    EncodeOptions

    Options controlling TOON encoding (TOON §13.1).

    ErrorKind

    Machine-matchable error classification for TOON decoding failures.

    FallbackFinding

    Path-level diagnostic explaining why a nested collection fell back from compact tabular encoding.

    FallbackReason

    Reason why a JSON array or object could not use TOON tabular/keyed-tabular compression.

    FieldNode

    Recursive schema node inside {...} header field lists (TOON §6 & §9.3), representing either a scalar leaf column or a nested uniform object group.

    FormatStats

    Per-format serialization and efficiency measurement.

    Span

    using @Waysyou/toon4/types { type Span }

    1-based source location span used in TOON diagnostics.

    StreamEvent

    Streaming decode event emitted by the recursive-descent TOON decoder.

    StructuralAudit

    Summary of tabular eligibility across a JSON payload.

    TextMetrics

    Byte, structural syntax overhead, and BPE-aware heuristic token metrics for a serialized text payload.

    VerdictKind

    High-level actionable recommendation from comparing TOON vs Compact JSON.

    analyze

    Evaluates value across Pretty JSON, Compact JSON, and TOON (comma, tab, pipe), returning byte/token comparisons, tabular coverage, and verdict.

    decode

    Decodes a TOON v4.4 document string into a Json value.

    decode_bytes

    Strictly validates UTF-8 bytes and decodes a TOON v4.4 document into Json.

    decode_stream

    Streams @types.StreamEvent values from a TOON v4.4 document string.

    decode_stream_lines

    Streams @types.StreamEvent values by pulling lines lazily from next_line.

    encode

    Encodes a Json value into a TOON v4.4 string.

    encode_lines

    Encodes a Json value into an array of TOON v4.4 lines.

    encode_to_lines

    fn encode_to_lines(value :
    Json
    , emit : (String) -> Unit, options? :
    EncodeOptions
    ) -> Unit raise
    EncodeError

    Streams TOON v4.4 lines for value directly to emit.

    format_decode_error

    fn format_decode_error(err :
    DecodeError
    ) -> String

    Formats a DecodeError with line/column span and source caret underline.

    format_encode_error

    fn format_encode_error(err :
    EncodeError
    ) -> String

    Formats an EncodeError for human display.

    measure_text

    fn measure_text(text : String) ->
    TextMetrics

    Measures exact UTF-8 bytes, syntax overhead bytes, and heuristic BPE tokens.

    spec_version

    fn spec_version() -> String

    Supported TOON specification version targeted by Waysyou/toon4.

    Source Files