#tokenizers-moonbit

Status: actively developed. See PROGRESS.md for the full capability matrix and roadmap.

中文文档：README.zh.md

#Why this project

• Runs everywhere MoonBit runs. One pure-MoonBit implementation compiles to wasm/wasm-gc/js/native. No FFI, no platform-specific binaries.
• Faithful to HuggingFace. Output is checked token-for-token against the Python tokenizers library for real models, including classic encoders, modern LLM tokenizers, coder tokenizers, multimodal tokenizers and embedding tokenizers.
• Loads tokenizer.json directly. No conversion step; use the same file as your Python or Transformers pipeline.
• Optional Hub download. The core loader stays all-backend/offline, while the hub package can download tokenizer.json on native/js and populate the same HuggingFace-style cache layout.

#Supported

• Models: BPE, byte-level BPE (with byte_fallback / fuse_unk / ignore_merges), WordPiece, Unigram, WordLevel.
• Pipeline: Normalizer → Pre-tokenizer → Model → Post-processor → Decoder, plus AddedVocabulary for special/added tokens such as <|endoftext|> and [MASK], including single_word, lstrip, rstrip and normalized flags.
• API: encode, encode_pair, encode_batch, decode, truncation and padding builders, token_to_id, id_to_token, get_vocab_size.

#Documentation

• Usage guide — loading, encode/decode, truncation, padding, batches
• API reference
• Supported components & limitations
• Migrating from HuggingFace
• Benchmarks
• 中文文档

#Quick start

moon new my-app && cd my-app
moon add howtomakeaname/tokenizers-moonbit

import {
  "howtomakeaname/tokenizers-moonbit/tokenizer",
}

fn main {
  try {
    // Load from a tokenizer.json string (backend-agnostic, no file IO):
    // let tok = @tokenizer.Tokenizer::from_str(json_text)
    // Or load from a file (uses moonbitlang/x/fs, available on all backends):
    let tok = @tokenizer.from_file("tokenizer.json")

    // Encode:
    let enc = tok.encode("Hello world")
    println(enc.ids)            // [Int]
    println(enc.tokens)         // [String]
    println(enc.attention_mask) // [Int]

    // Decode:
    let text = tok.decode(enc.ids, skip_special_tokens=true)
    println(text)
  } catch {
    e => println("failed: \{e.message()}")
  }
}

async fn main {
  try {
    let tok = @hub.from_pretrained("bert-base-uncased")

    // Use a HuggingFace-compatible mirror when needed:
    let tok2 = @hub.from_pretrained(
      "bert-base-uncased",
      options=@hub.HubDownloadOptions::new(endpoint="https://hf-mirror.com"),
    )
    println(tok2.get_vocab_size())
  } catch {
    e => println("failed: \{e.message()}")
  }
}

#Migrating from HuggingFace

#Building & testing

export PATH="$HOME/.moon/bin:$PATH"
moon test                    # default backend (wasm-gc)
moon test --target native    # also: wasm, wasm-gc, js

# download model fixtures (classic + modern matrix; gated/renamed models skip)
python3 scripts/fetch_models.py

# generate expected outputs with Python tokenizers
pip install tokenizers
python3 scripts/gen_parity.py

moon test --target native

#License