TOON v4.4 codec, line/column diagnostic engine, and JSON vs TOON token-efficiency advisor in MoonBit
| Capability / Spec Clause | oliveagle/toon-mbit (audited) | Waysyou/toon4 |
|---|---|---|
| Toolchain manifest (moon.mod / moon.pkg) | Uses legacy moon.mod + moon.pkg.json (rejected by moon 0.1.20261117) | Modern moon.mod + moon.pkg DSL; zero warnings (--deny-warn) across wasm, wasm-gc, js, native |
| TOON 4.4 §9.5 Keyed Tabular (users[2:]{name,score}:) | Fails at src/decoder.mbt:273-295, 360-369 (: inside [2:] treated as key-value colon / illegal symbol) | Full encode, decode, nested field groups, [0:], delimiter support, and strict/non-strict rules |
| §9.3 Nested Field Groups (regions[2:]{name,geo{lat,lon}}:) | Not supported | Recursive FieldNode ADT (Leaf / Group) in both encoder and decoder |
| §11 Tab Delimiter Header (items[2\t]:) | src/decoder.mbt:299 .trim() strips trailing \t before parsing bracket delimiter | Preserves exact HTAB and | header delimiters; validates {fields} delimiter consistency |
| §5.1 Full-line Comments (# ...) | Not stripped | Pre-pass strips space-indented # lines before blank-line, quote, and indent checks |
| §9.1 / §9.2 Empty Arrays | Encodes deprecated [0]: / key[0]: (src/encoder.mbt:194-209); decodes [] as string | Emits [] / key: [] / - [0]:; decodes [] as array in root/field/list-item positions and "[]" inside cells |
| §4 Strict Numbers (1e5, 05, +1, 1.) | 05 parsed as integer 5; 1e5 fails parse_double | Exact §4 DFA (1e5 -> 100000, 05 / +1 / 1. / 0x10 stay strings, -0.0 -> 0) |
| §7.1 Escapes & Quotes (\uXXXX, "a" "b") | Rejects \uXXXX; unquote_string skips interior quote check ("a" "b" passes) | Full \uXXXX + surrogate rejection; strict closing-quote boundary check |
| §14 Strict vs 5 Non-Strict Recoveries | Checks strict only on inline array counts (src/decoder.mbt:401) | Full strict enforcement + all 5 §14.4 non-strict recoveries (preserving key insertion order via Map) |
| UTF-16 Emoji Index Safety | Mixes Char iterator offsets with UTF-16 slices (src/decoder.mbt:593-612) | Scans UTF-16 code units consistently; maps offsets to 1-based Unicode scalar columns for carets |
| Error Diagnostics | ParseError { line, message } hardcodes line: 0 in all 6 header errors | DecodeError with ErrorKind, Span(line, column, length), source line, and caret formatting |
DecodeError[E_WIDTH] at line 3, column 10: expected 3 keyed entry cells, but got 2
3 | WH-SZ: Shenzhen,1
| ^^^^^^^^^^DecodeError[E_INDENT] at line 2, column 1: tabs are not allowed in indentation in strict mode
2 | →b: 1
| ^TOON v4.4 Cost & Token-Efficiency Analysis
------------------------------------------------------------------------
JSON (pretty) ████████████████████ ~4835 tok (14113 B, 7388 syntax B, +79.3% vs compact)
JSON (compact) ███████████░░░░░░░░░ ~2696 tok (9353 B, 2628 syntax B, 0.0% vs compact)
* TOON (comma) ███████░░░░░░░░░░░░░ ~1584 tok (4139 B, 686 syntax B, -41.2% vs compact)
TOON (tab) ███████░░░░░░░░░░░░░ ~1587 tok (4141 B, 688 syntax B, -41.1% vs compact)
TOON (pipe) ███████░░░░░░░░░░░░░ ~1587 tok (4141 B, 688 syntax B, -41.1% vs compact)
------------------------------------------------------------------------
Tabular coverage: 100.0% (tabular_items=83, fallback_items=0, array_blocks=1, keyed_blocks=1)
Verdict [RECOMMEND_TOON]: Recommend TOON (comma): -41.2% estimated tokens vs compact JSON (-67.2% vs pretty JSON), tabular coverage 100.0%.Tabular coverage: 48.6% (tabular_items=18, fallback_items=19, array_blocks=0, keyed_blocks=9)
Fallback paths:
- tests (14 items): element [1] adds extra key 'note' absent in [0]; falls back to list form
- tests[7].input.config (1 items): object has only 1 entry (< 2 minimum for TOON 4.4 keyed tabular '[N:]')
- tests[11].input.items (2 items): element [1] is missing key 'config' present in [0]; falls back to list form
- tests[12].input.items (2 items): contains non-object, empty-object, or nested-array elements; uses list form ('- ')
Verdict [KEEP_COMPACT_JSON]: Keep compact JSON: TOON (comma) adds +19.1% estimated tokens vs compact JSON (tabular coverage 48.6%; 4 non-tabular path(s)).| Benchmark | native (mean ± σ) | wasm / moonrun (mean ± σ) |
|---|---|---|
| encode_inventory_200 (200 SKU rows, 8 cols) | 411.67 µs ± 10.37 µs | 479.54 µs ± 16.71 µs |
| decode_inventory_200 (200 SKU rows, 8 cols) | 1.07 ms ± 78.63 µs | 1.09 ms ± 28.40 µs |
| encode_keyed_flags_80 (80 keyed entries, [80:]) | 145.00 µs ± 2.32 µs | 144.37 µs ± 2.86 µs |
| decode_keyed_flags_80 (80 keyed entries, [80:]) | 274.56 µs ± 10.65 µs | 281.96 µs ± 6.96 µs |
| analyze_inventory_200 (3 TOON + 2 JSON + audit) | 2.94 ms ± 129.90 µs | 4.25 ms ± 295.95 µs |
| analyze_nested_orders_50 (50 nested orders) | 1.59 ms ± 42.72 µs | 2.30 ms ± 80.30 µs |
import {
"Waysyou/toon4",
}let doc = Json::object(Map([
("users", Json::object(Map([
("alice", Json::object(Map([("name", Json::string("Alice")), ("score", Json::number(95.0))]))),
("bob", Json::object(Map([("name", Json::string("Bob")), ("score", Json::number(88.0))]))),
]))),
]))
let toon_text = @toon4.encode(doc)
let round_trip = @toon4.decode(toon_text)
let report = @toon4.analyze(doc)
println(report.render_report())# Compare JSON vs TOON efficiency and inspect non-tabular fallback paths
moon run --target native cmd/toon4 -- analyze data.json
moon run --target native cmd/toon4 -- analyze --json data.json
# Encode JSON to TOON v4.4 (comma | tab | pipe) with stats to stderr
moon run --target native cmd/toon4 -- encode -d pipe --stats data.json
# Validate TOON with pinpoint line/column caret diagnostics
moon run --target native cmd/toon4 -- check payload.toon
# Decode TOON (strict by default, or --no-strict for §14.4 recoveries)
moon run --target native cmd/toon4 -- decode payload.toonmoon check --deny-warn --target all
moon test --target native
moon test --target wasm
moon test --target wasm-gc
moon test --target js
moon bench -p Waysyou/toon4/benchmarks --target nativefn decode_stream(input : String, on_event : (StreamEvent) -> Unit raise DecodeError, options? : DecodeOptions) -> Unit raise DecodeErrorfn decode_stream_lines(next_line : () -> String?, on_event : (StreamEvent) -> Unit raise DecodeError, options? : DecodeOptions) -> Unit raise DecodeErrorfn encode_to_lines(value : Json, emit : (String) -> Unit, options? : EncodeOptions) -> Unit raise EncodeErrorfn spec_version() -> StringInstall
Download zipTOON v4.4 codec, line/column diagnostic engine, and JSON vs TOON token-efficiency advisor in MoonBit