🥮 mooncakes.io

    moontoken

    Tokenizers for MoonBit: byte-level BPE and SentencePiece.

    tokenizer
    tokenizers
    tokenization
    token
    count-tokens
    bpe
    byte-pair-encoding
    byte-level-bpe
    sentencepiece
    unigram
    tiktoken
    cl100k
    cl100k_base
    o200k
    o200k_base
    r50k_base
    p50k_base
    tokenizer-json
    vocabulary
    vocab
    merges
    encode
    decode
    special-tokens
    byte-fallback
    pre-tokenizer
    normalization
    nlp
    llm
    prompt
    context-window
    moonbit
    Download zip
    Version
    0.1.1
    License
    Apache-2.0
    Repository
    Last updated
    13 hours ago
    Downloads
    9

    #moontoken

    Text into tokens and back, over a vocabulary you supply.

    A tokenizer is three things: a way to cut text into the words a model was trained on, a vocabulary that says what each of those words costs, and a rule for the tokens a model reserves. This library implements all three for the two families in use today, and takes the vocabulary as an argument. It never opens a file of its own accord, because where a vocabulary comes from is the format's business: a GGUF file is moongguf's, a tokenizer.json and a SentencePiece model are read here, and a published rank table is a string.

    let tokenizer = @moontoken.Tokenizer::cl100k(ranks~)
    let ids = tokenizer.encode("the world and the thing")

    text       the world and the thing
tokens     8
ids        258 262 261 32 258 32 256 260
bytes      23
decoded    the world and the thing
count      8
vocabulary 263

    Encoding and decoding round-trip: text that goes in comes back out, including the bytes that are not valid text on their own, because a byte-level encoding spells those out rather than dropping them.

    #What it covers

    PackageWhat it is
    moontokenThe tokenizer: encode, encode_ordinary, decode, count, and the ways to build one
    splitThe pre-tokenizers of the published encodings, written as scanners
    vocabA rank table, from the base64 file OpenAI publishes or from pairs
    bpeByte-pair merging: the byte-level model, and the word-level one a tokenizer.json names
    pieceThe SentencePiece model file: both trainer types, and the trie it compiles
    normalizerThe rule table a model file carries, and the operations a tokenizer.json lists
    specialThe tokens a model reserves, and what may happen to them
    refusedThe one error shape the whole library raises

    Three families of vocabulary, each with its own entry point:

    • Byte-level BPE. Tokenizer::cl100k, ::o200k, ::r50k, ::p50k_edit for the published encodings, or ::byte_bpe with a split and a policy you choose.
    • SentencePiece. Tokenizer::of_proto reads a .model file, unigram or BPE. The trie, the normalisation rules, the byte fallback and the user-defined symbols all come from the file; nothing is assumed.
    • tokenizer.json. Tokenizer::parse reads a BPE or unigram vocabulary with its merges, normaliser, pre-tokenizer and added tokens. Anything the file asks for that this reader does not implement is refused by name rather than approximated.

    #Reserved tokens

    A model reserves a handful of spellings that stand for structure rather than text. Whether text that happens to contain <|endoftext|> is taken as the token, taken as text, or refused is the caller's decision, never the library's:

    tokenizer.encode(text, special=@special.Refuse)   // refuse it
    tokenizer.encode(text, special=@special.Allow)    // take it as the token
    tokenizer.encode(text, special=@special.Ignore)   // encode it as text
    tokenizer.encode(text, special=@special.Only(["<|endoftext|>"]))

    When one spelling is a prefix of another, the longest one wins. The reference implementation takes whichever a hash map happens to yield first, which is not a definition a tokenizer can have; the test that pins the difference is the longest reserved token wins.

    #Conformance

    Two layers, both in CI.

    Vectors. Generated from the references by scripts/gen_vectors.py, with the versions pinned: tiktoken 0.14.0 and sentencepiece 0.2.3.

    WhatRowsPassing
    cl100k_base and o200k_base, over a 600-entry slice of each published table8484
    The reserved tokens, under each policy66
    SentencePiece unigram, BPE, and unigram without the dummy prefix258258
    Total348348

    The models are trained from a corpus the script writes and the trainer is pinned to one thread, so a run is reproducible byte for byte; CI re-runs the generator and requires the working tree to stay clean. A vector that no longer matches its reference is worse than no vector at all.

    Live differential. scripts/diff_reference.py asks the references again, now, for 242 records: the texts the vectors use plus 200 random strings over the characters the patterns treat specially, astral planes included. Every record is compared in both directions through src/examples/03-conformance.

    ReferenceEncodersComparisonsAgreeing
    tiktoken 0.14.0419361936
    sentencepiece 0.2.3314521452
    tokenizers 0.23.12968968
    Total943564356

    Three references rather than one, because none of them covers all three formats, and a format read by one implementation only is a format that drifts.

    moon build --target native
pip install tiktoken==0.14.0 sentencepiece==0.2.3 tokenizers==0.23.1
python scripts/diff_reference.py --prepare

    Mutation. Two deliberate breaks, to check that the tests are load-bearing. Taking the highest rank instead of the lowest in the merge loop fails three tests: cl100k_base agrees with the reference vectors, o200k_base agrees withthe reference vectors, and reserved tokens follow the policy. Taking the first reserved spelling that matches instead of the longest fails two: the longest reserved token wins and the longest spelling at a position isfound. Both were reverted.

    No test is skipped, so there is nothing to report as not passing.

    #Deliberately elsewhere

    • Where a vocabulary comes from. A GGUF file, a Hugging Face download, an OpenAI table on disk. This library parses the formats and takes the bytes; fetching them is the caller's.
    • Training. sentencepiece and tokenizers train vocabularies; this reads them. The scripts under scripts/ train small ones, and they are test fixtures, not an API.

    #Install

    moon add moonbitstack/moontoken

    #Licence

    Apache-2.0. See LICENSE.

    Refused

    source

    Why an input was refused.

    Malformed carries the offset, Absent the name of what was missing, and Missing the id that is not in the vocabulary, so that a caller reading a table of a hundred thousand entries can find the one that broke.

    Model

    source
    pub(open) trait Model {
      fn encode(Self, StringView) -> Array[Int] raise 
    Refused
      fn decode_bytes(Self, Array[Int]) -> Bytes raise 
    Refused
      fn decode(Self, Array[Int]) -> String raise 
    Refused
      fn token_bytes(Self, Int) -> Bytes?
      fn length(Self) -> Int
    }

    What a tokenizer needs from a segmentation algorithm.

    The four models this library ships are byte-level byte-pair encoding, the two sentence-piece models, and the pair-merge model of a tokenizer.json file. A caller with a fifth implements this and gets the whole surface: text in, tokens out, and the special tokens around them.

    Tokenizer

    source

    A tokenizer: a model, the tokens it reserves, and what may happen to them.

    Text goes in and tokens come out, and the tokens come back as the same text, including the bytes that are not text on their own. The vocabulary is the caller's: this library reads one it is handed and never opens a file.

    Tokenizer::byte_bpe

    source

    A tokenizer over a byte-level byte-pair table.

    Tokenizer::cl100k

    source

    The tokens of cl100k_base, given the table of ranks that encoding uses.

    Tokenizer::count

    source

    How many tokens text holds.

    This is encode with every reserved token recognised, which is the number a caller budgeting a context window wants: the reserved spellings cost their one token rather than being taken apart as text.

    Tokenizer::decode

    source
    fn Tokenizer::decode(self : Tokenizer, ids : Array[Int], reserved? : Bool) -> String raise 
    Refused

    The text a list of tokens stands for, with U+FFFD standing in for the byte sequences that are not text.

    Tokenizer::decode_bytes

    source
    fn Tokenizer::decode_bytes(self : Tokenizer, ids : Array[Int], reserved? : Bool) -> Bytes raise 
    Refused

    The bytes a list of tokens stands for.

    Tokenizer::encode

    source

    The tokens text holds.

    special overrides the policy this tokenizer was built with, for a caller that knows this one string came from a model that emits the reserved spelling itself.

    Tokenizer::encode_ordinary

    source
    fn Tokenizer::encode_ordinary(self : Tokenizer, text : StringView) -> Array[Int] raise 
    Refused

    The tokens of text that holds no reserved token, whatever the policy says.

    Tokenizer::length

    source
    fn Tokenizer::length(self : Tokenizer) -> Int

    How many tokens the vocabulary has, reserved ones not counted.

    Tokenizer::new

    source

    A tokenizer over any model.

    Tokenizer::o200k

    source

    The tokens of o200k_base, given the table of ranks that encoding uses.

    Tokenizer::of

    source

    A tokenizer from the tokenizer.json a model distribution ships.

    The file describes a vocabulary, the merges over it, the normalisation, the cut, and the tokens the model reserves. Everything in it that this reader does not implement is refused by name rather than approximated.

    Tokenizer::of_proto

    source

    A tokenizer from a sentence-piece model file.

    Tokenizer::p50k_edit

    source

    The tokens of p50k_edit, given the table of ranks that encoding uses.

    Tokenizer::parse

    source

    A tokenizer from the text of a tokenizer.json file.

    Tokenizer::piece_bpe

    source

    A tokenizer over a sentence-piece vocabulary that segments by merging neighbouring symbols.

    Tokenizer::policy

    source

    What this tokenizer does with the text that holds a reserved token.

    Tokenizer::r50k

    source

    The tokens of r50k_base, given the table of ranks that encoding uses.

    Tokenizer::token_bytes

    source
    fn Tokenizer::token_bytes(self : Tokenizer, id : Int) -> Bytes?

    The bytes one token stands for.

    Tokenizer::unigram

    source

    A tokenizer over a sentence-piece vocabulary that segments by the most likely sequence of pieces.

    Tokenizer::with_policy

    source

    A tokenizer with a different answer for the text that holds a reserved token.

    Source Files

    Powered by MoonBit

    Site sourceReport issueModulesSkillsCLIBuild queueStatistics

    © 2026 mooncakes.io