🥮 mooncakes.io

    moontoken

    Tokenizers for MoonBit: byte-level BPE and SentencePiece.

    tokenizer
    tokenizers
    tokenization
    token
    count-tokens
    bpe
    byte-pair-encoding
    byte-level-bpe
    sentencepiece
    unigram
    tiktoken
    cl100k
    cl100k_base
    o200k
    o200k_base
    o200k_harmony
    harmony
    gpt-oss
    chat-template
    r50k_base
    p50k_base
    p50k_edit
    gpt2
    tokenizer-json
    vocabulary
    vocab
    merges
    encode
    decode
    special-tokens
    byte-fallback
    pre-tokenizer
    normalization
    nlp
    llm
    prompt
    context-window
    moonbit
    Download zip
    Version
    0.2.0
    License
    Apache-2.0
    Repository
    Last updated
    10 hours ago
    Downloads
    257

    #moontoken

    Text into tokens and back, over a vocabulary you supply.

    A tokenizer is three things: a way to cut text into the words a model was trained on, a vocabulary that says what each of those words costs, and a rule for the tokens a model reserves. This library implements all three for the two families in use today, and takes the vocabulary as an argument. It never opens a file of its own accord, because where a vocabulary comes from is the format's business: a GGUF file is moongguf's, a tokenizer.json and a SentencePiece model are read here, and a published rank table is a string.

    /// `table` is the text of a published `cl100k_base.tiktoken` file.
    fn ids_of(table : String, text : String) -> Array[Int] raise {
      let ranks = @vocab.Ranks::from_tiktoken(table)
      let tokenizer = @moontoken.Tokenizer::cl100k(ranks~)
      tokenizer.encode(text)
    }

    text       the world and the thing
tokens     8
ids        258 262 261 32 258 32 256 260
bytes      23
decoded    the world and the thing
count      8
vocabulary 263

    Encoding and decoding round-trip: text that goes in comes back out, including the bytes that are not valid text on their own, because a byte-level encoding spells those out rather than dropping them.

    #The interface

    One type carries the vocabulary and the rules; everything else is an argument to it.

    CallWhat it does
    Tokenizer::cl100k(ranks~)cl100k_base: its split, its reserved tokens, over the table you supply. ::o200k, ::o200k_harmony, ::r50k and ::p50k_edit are the same call for the other published encodings; gpt2 and p50k_base are ::r50k's rules over their own tables.
    Tokenizer::byte_bpe(ranks~, split~)A table with a split of your own: any rule in split, or a sequence of them.
    Tokenizer::of_proto(bytes)A SentencePiece .model file, unigram or BPE, read whole.
    Tokenizer::parse(text)A tokenizer.json document. ::of takes the same thing as a Json you already parsed.
    encode(text)The ids. special= overrides the policy for this one call.
    encode_ordinary(text)The ids of text that holds no reserved token, whatever the policy says.
    count(text)How many ids the text costs when every reserved token is recognised — the number to budget a context window with.
    decode(ids)The text, with U+FFFD standing in for the byte sequences that are not text.
    decode_bytes(ids)The bytes, exact. reserved=false drops the reserved tokens instead of spelling them.
    token_bytes(id)The bytes one id stands for, reserved ids included.
    length()How many entries the vocabulary has, reserved ones not counted.

    Every fallible call raises the one error the library has, Refused, which is moonbitstack/moontoken/refused re-exported. Match on it rather than printing it: Malformed(what, at) is a format that does not parse, Absent(name) something a file should have carried and did not, Unsupported(what) a format this reader does not implement, Missing(id) an id the vocabulary does not have, and Disallowed(what, at) a reserved spelling a policy refused.

    #What it covers

    PackageWhat it is
    moontokenThe tokenizer: encode, encode_ordinary, decode, count, and the ways to build one
    splitThe pre-tokenizers of the published encodings, written as scanners
    vocabA rank table, from the base64 file OpenAI publishes or from pairs
    bpeByte-pair merging: the byte-level model, and the word-level one a tokenizer.json names
    pieceThe SentencePiece model file: both trainer types, and the trie it compiles
    normalizerThe rule table a model file carries, and the operations a tokenizer.json lists
    specialThe tokens a model reserves, and what may happen to them
    refusedThe one error shape the whole library raises

    Three families of vocabulary, each with its own entry point:

    • Byte-level BPE. Tokenizer::cl100k, ::o200k, ::o200k_harmony, ::r50k, ::p50k_edit for the published encodings, or ::byte_bpe with a split and a policy you choose. gpt2 and p50k_base share r50k's split and its one reserved token, so they read through ::r50k too.
    • SentencePiece. Tokenizer::of_proto reads a .model file, unigram or BPE. The trie, the normalisation rules, the byte fallback and the user-defined symbols all come from the file; nothing is assumed.
    • tokenizer.json. Tokenizer::parse reads a BPE or unigram vocabulary with its merges, normaliser, pre-tokenizer and added tokens. Anything the file asks for that this reader does not implement is refused by name rather than approximated.

    #Reserved tokens

    A model reserves a handful of spellings that stand for structure rather than text. Whether text that happens to contain <|endoftext|> is taken as the token, taken as text, or refused is the caller's decision, never the library's:

    fn four_ways(tokenizer : @moontoken.Tokenizer, text : String) -> Array[Array[Int]] raise {
      [
        tokenizer.encode(text, special=@special.Refuse), // refuse it
        tokenizer.encode(text, special=@special.Allow), // take it as the token
        tokenizer.encode(text, special=@special.Ignore), // encode it as text
        tokenizer.encode(text, special=@special.Only(["<|endoftext|>"])),
      ]
    }

    When one spelling is a prefix of another, the longest one wins. The reference implementation takes whichever a hash map happens to yield first, which is not a definition a tokenizer can have; the test that pins the difference is the longest reserved token wins.

    #Conformance

    Two layers, both in CI.

    Vectors. Generated from the references by scripts/gen_vectors.py, with the versions pinned: tiktoken 0.14.0 and sentencepiece 0.2.3.

    WhatRowsPassing
    cl100k_base and o200k_base, over a 600-entry slice of each published table8484
    The reserved tokens, under each policy66
    SentencePiece unigram, BPE, and unigram without the dummy prefix258258
    Total348348

    The models are trained from a corpus the script writes and the trainer is pinned to one thread, so a run is reproducible byte for byte; CI re-runs the generator and requires the working tree to stay clean. A vector that no longer matches its reference is worse than no vector at all.

    Live differential. scripts/diff_reference.py asks the references again, now, for 249 records: the texts the vectors use, the control words of a chat format, and 200 random strings over the characters the patterns treat specially, astral planes included. Every record is compared in both directions through the bench module under integration/.

    ReferenceEncodersComparisonsAgreeing
    tiktoken 0.14.0629882988
    sentencepiece 0.2.3314941494
    tokenizers 0.23.12996996
    Total1154785478

    Three references rather than one, because none of them covers all three formats, and a format read by one implementation only is a format that drifts.

    moon build --target native
pip install tiktoken==0.14.0 sentencepiece==0.2.3 tokenizers==0.23.1
python scripts/diff_reference.py --prepare

    Mutation. Five deliberate breaks, to check that the tests are load-bearing; each was made, run, and reverted. Taking the highest rank instead of the lowest in the merge loop fails three tests: cl100k_base agrees with the referencevectors, o200k_base agrees with the reference vectors, and reserved tokensfollow the policy. Taking the first reserved spelling that matches instead of the longest fails four: the policy a tokenizer carries can be replaced, only the spellings named are taken as tokens, the longest reserved tokenwins, and the longest spelling at a position is found. Requiring base64 padding fails a token without its padding is read. Letting the second spelling of an id overwrite the first fails the harmony encoding reserves its controlwords and placeholders. Reading the document as a lenient dialect fails text that is not one strict JSON document is refused.

    No test is skipped, so there is nothing to report as not passing.

    #Deliberately elsewhere

    • Where a vocabulary comes from. A GGUF file, a Hugging Face download, an OpenAI table on disk. This library parses the formats and takes the bytes; fetching them is the caller's.
    • Training. sentencepiece and tokenizers train vocabularies; this reads them. The scripts under scripts/ train small ones, and they are test fixtures, not an API.

    #Install

    moon add moonbitstack/moontoken

    #Licence

    Apache-2.0. See LICENSE.

    Refused

    source

    Why an input was refused.

    Malformed carries the offset, Absent the name of what was missing, and Missing the id that is not in the vocabulary, so that a caller reading a table of a hundred thousand entries can find the one that broke.

    Model

    source
    pub(open) trait Model {
      fn encode(Self, StringView) -> Array[Int] raise 
    Refused
      fn decode_bytes(Self, Array[Int]) -> Bytes raise 
    Refused
      fn decode(Self, Array[Int]) -> String raise 
    Refused
      fn token_bytes(Self, Int) -> Bytes?
      fn length(Self) -> Int
    }

    What a tokenizer needs from a segmentation algorithm.

    The four models this library ships are byte-level byte-pair encoding, the two sentence-piece models, and the pair-merge model of a tokenizer.json file. A caller with a fifth implements this and gets the whole surface: text in, tokens out, and the special tokens around them.

    Tokenizer

    source

    A tokenizer: a model, the tokens it reserves, and what may happen to them.

    Text goes in and tokens come out, and the tokens come back as the same text, including the bytes that are not text on their own. The vocabulary is the caller's: this library reads one it is handed and never opens a file.

    Tokenizer::byte_bpe

    source

    A tokenizer over a byte-level byte-pair table.

    Tokenizer::cl100k

    source

    The tokens of cl100k_base, given the table of ranks that encoding uses.

    Tokenizer::count

    source

    How many tokens text holds.

    This is encode with every reserved token recognised, which is the number a caller budgeting a context window wants: the reserved spellings cost their one token rather than being taken apart as text.

    Tokenizer::decode

    source
    fn Tokenizer::decode(self : Tokenizer, ids : Array[Int], reserved? : Bool) -> String raise 
    Refused

    The text a list of tokens stands for, with U+FFFD standing in for the byte sequences that are not text.

    Tokenizer::decode_bytes

    source
    fn Tokenizer::decode_bytes(self : Tokenizer, ids : Array[Int], reserved? : Bool) -> Bytes raise 
    Refused

    The bytes a list of tokens stands for.

    Tokenizer::encode

    source

    The tokens text holds.

    special overrides the policy this tokenizer was built with, for a caller that knows this one string came from a model that emits the reserved spelling itself.

    Tokenizer::encode_ordinary

    source
    fn Tokenizer::encode_ordinary(self : Tokenizer, text : StringView) -> Array[Int] raise 
    Refused

    The tokens of text that holds no reserved token, whatever the policy says.

    Tokenizer::length

    source
    fn Tokenizer::length(self : Tokenizer) -> Int

    How many tokens the vocabulary has, reserved ones not counted.

    Tokenizer::new

    source

    A tokenizer over any model.

    Tokenizer::o200k

    source

    The tokens of o200k_base, given the table of ranks that encoding uses.

    Tokenizer::o200k_harmony

    source

    The tokens of o200k_harmony, given the table of ranks o200k_base publishes: the same split and the same table, with the control words the harmony chat format writes.

    Tokenizer::of

    source

    A tokenizer from the tokenizer.json a model distribution ships.

    The file describes a vocabulary, the merges over it, the normalisation, the cut, and the tokens the model reserves. Everything in it that this reader does not implement is refused by name rather than approximated.

    Tokenizer::of_proto

    source

    A tokenizer from a sentence-piece model file.

    Tokenizer::p50k_edit

    source

    The tokens of p50k_edit, given the table of ranks that encoding uses.

    Tokenizer::parse

    source

    A tokenizer from the text of a tokenizer.json file.

    The text is read as strict RFC 8259: a trailing comma, a comment or a control character inside a string is refused where it is, rather than being quietly accepted and changing what the vocabulary ends up holding.

    Tokenizer::piece_bpe

    source

    A tokenizer over a sentence-piece vocabulary that segments by merging neighbouring symbols.

    Tokenizer::policy

    source

    What this tokenizer does with the text that holds a reserved token.

    Tokenizer::r50k

    source

    The tokens of r50k_base, given the table of ranks that encoding uses.

    gpt2 and p50k_base are the same split and the same one reserved token over their own tables, so they read through this constructor too.

    Tokenizer::token_bytes

    source
    fn Tokenizer::token_bytes(self : Tokenizer, id : Int) -> Bytes?

    The bytes one token stands for.

    Tokenizer::unigram

    source

    A tokenizer over a sentence-piece vocabulary that segments by the most likely sequence of pieces.

    Tokenizer::with_policy

    source

    A tokenizer with a different answer for the text that holds a reserved token.

    Source Files