#MoonMojibake — Encoding Forensics & Mojibake Repair for MoonBit

1. Detection — it answers "what encoding is this file?" without being told. It decodes the same bytes under every code page it knows, scores how much each result looks like a real language, and ranks them with a confidence.
2. Repair — it answers "this text is mojibake, what happened?" It searches for the re-encode/decode step that turns the garbage back into text, applies it, and repeats, because files really do travel through more than one broken tool. The result comes with the chain that produced it, e.g. windows-1252 -> utf-8.

#Status

#Quick start

#As a command

moon build --target js
alias mm="node _build/js/debug/build/cmd/moonmojibake/moonmojibake.js"

mm detect samples/gbk.txt

samples/gbk.txt (36 bytes)

  #  encoding        score  conf  errors  sample
  1  gbk                86    51       0  我们把这件事说清楚，然后开始写代码。
  2  euc-kr             69     0       0  乖쳬겉侶숭慤綱헌뇝，횔빈역迦畇덜쯤。
  3  euc-jp             66     0       2  厘断委宸周並傍賠萱��隼朔蝕兵亟旗鷹。
  4  big5               54     0       2  扂蠅參涴璃岈佽ь奠ㄛ�遣騜羌標斐�鎢﹝
  ...
 10  utf-8               3     0      19  ���ǰ������˵�����Ȼ��ʼд���
 12  utf-32be            0     0       9  ���������

guess: gbk  (confidence 51%, score 86)

mm repair samples/moji_latin1.txt

samples/moji_latin1.txt
score 40 -> 86
chain: latin1 -> utf-8

我们把这件事说清楚，然后开始写代码。

mm convert samples/utf8.txt --to gbk -o out.bin
mm detect --text "中文测试"

mm scan samples -r

scan samples (8 files)

  encoding   files
  big5           1
  euc-jp         1
  euc-kr         1
  gbk            1
  utf-8          3
  shift_jis      1

  samples/big5.txt                    big5
  samples/eucjp.txt                   euc-jp
  samples/euckr.txt                   euc-kr
  samples/gbk.txt                     gbk
  samples/moji_cp1252.txt             utf-8
  samples/moji_latin1.txt             utf-8
  samples/sjis.txt                    shift_jis
  samples/utf8.txt                    utf-8

mm fix samples -r            # dry run: what would change
mm fix samples -r --write    # actually rewrite them as UTF-8

fix samples (recursive) (8 files, dry run)

  path                                    read as       change
  samples/big5.txt                        big5          transcode
  samples/eucjp.txt                       euc-jp        transcode
  samples/euckr.txt                       euc-kr        transcode
  samples/gbk.txt                         gbk           transcode
  samples/moji_cp1252.txt                 utf-8         repair: windows-1252 -> utf-8  45 -> 90
  samples/moji_latin1.txt                 utf-8         repair: latin1 -> utf-8  40 -> 86
  samples/sjis.txt                        shift_jis     transcode

7 of 8 text files would change; nothing written. Re-run with --write to apply.

mm detect samples/gbk.txt --json | jq -r '.guess, .confidence'
mm repair samples/moji_latin1.txt --json | jq -r '.chain[0].wrong + " -> " + .chain[0].right'
mm scan . -r --json | jq -r '.entries[] | select(.binary or .charset != "utf-8") | .path'

#In CI

mm scan . -r --fail-if-not utf-8

error: docs/legacy.txt reads as gbk, expected utf-8

#As a library

// Detect: which code page produced these bytes?
let rows = @mojibake.detect(bytes)
let best = rows[0]
println(best.charset().name())      // "gbk"
println(best.confidence())          // 0..100
println(best.text())                // the decoded text

// Repair: undo the accident layer by layer.
let fixed = @mojibake.repair(mojibake_text)
println(fixed.text())               // the recovered text
println(@mojibake.render_chain(fixed))  // "windows-1252 -> utf-8"

// Direct control when you already know both ends.
let decoded = @mojibake.decode(bytes, @mojibake.Charset::Big5)
let encoded = @mojibake.encode("繁體中文", @mojibake.Charset::Big5)

#How detection works

• Valid multi-byte UTF-8 is close to proof. A legacy code page forming a well-shaped UTF-8 sequence across a whole file is astronomically unlikely, while UTF-8 read as a legacy code page happens daily.
• BOM-less UTF-16 has to prove itself by showing the alternating-zero pattern. Without a BOM, any even-length stream decodes into something, and arbitrary byte pairs land in Hangul, where they look deceptively plausible.

#How repair works

#What it covers

#Testing

moon test                        # 67 unit tests
moon build --target js
python scripts/e2e_check.py      # 257 cross-checks against CPython
python scripts/space_check.py    # 127 287 exhaustive checks, both directions

#Limitations, stated plainly

• The CLI reads and writes files only on the js backend; on other backends the library still works, and --text still runs, but file IO is not available. MoonBit core has no portable filesystem API.
• Some byte strings are genuinely ambiguous, and the tool says so rather than guessing. A kana-only EUC-JP file lives in the JIS X 0208 rows that GBK keeps for the same characters, so both names decode it to identical text; detect prints ambiguous — gbk and euc-jp …, --json carries ambiguous and tied, and --fail-if-not accepts either name. Pure ASCII is the same situation by construction: every ASCII-compatible page agrees.
• Western text shorter than a sentence is ambiguous for the same reason: Latin-1 and windows-1252 are identical for anything that does not use 0x80–0x9F, and both look plausible under several code pages.
• BOM-less UTF-16 holding CJK text cannot be proven and is deliberately discounted; UTF-16 with a BOM is detected exactly.
• Healing is bounded by information loss. Once a byte became U+FFFD, the original is gone.

#Documentation

• docs/USAGE.md — CLI and library reference
• docs/FORMAT_NOTES.md — table layout, scoring weights, design reasoning

#License