🥮 mooncakes.io

    moonbitstack/moontoken/bpe does not have a README file

    ByteBpe

    source
    pub struct ByteBpe {
      ranks : 
    Ranks
      split : 
    Split
      cache : Map[String, Array[Int]]
      options : Options
    }

    A byte-level byte-pair encoder: text is cut into words, each word is taken as UTF-8 bytes, and the merge table turns those bytes into tokens.

    Every byte has a token of its own, which is what makes the encoding total: any text, valid or not, has an encoding of at least one token per byte.

    ByteBpe::decode

    source
    fn ByteBpe::decode(self : ByteBpe, ids : ArrayView[Int]) -> String raise 
    Refused

    The text a list of tokens stands for, with U+FFFD standing in for the byte sequences that are not text.

    ByteBpe::decode_bytes

    source
    fn ByteBpe::decode_bytes(self : ByteBpe, ids : ArrayView[Int]) -> Bytes raise 
    Refused

    The bytes a list of tokens stands for.

    The result need not be valid UTF-8: a caller that cut a character in half gets the two halves back, which is what makes re-encoding exact.

    ByteBpe::encode

    source
    fn ByteBpe::encode(self : ByteBpe, text : StringView) -> Array[Int] raise 
    Refused

    The tokens of text.

    ByteBpe::encode_word

    source
    fn ByteBpe::encode_word(self : ByteBpe, word : StringView) -> Array[Int] raise 
    Refused

    The tokens of one word, without cutting it first.

    ByteBpe::length

    source
    fn ByteBpe::length(self : ByteBpe) -> Int

    How many tokens the vocabulary has.

    ByteBpe::new

    source

    Builds an encoder over a table of ranks.

    Raises when a single byte has no token, because such a table cannot encode text that happens to contain that byte, and finding out mid-sentence is worse than finding out here.

    ByteBpe::token_bytes

    source
    fn ByteBpe::token_bytes(self : ByteBpe, id : Int) -> Bytes?

    The bytes one token stands for.

    Decoding

    source
    pub(all) enum Decoding {
      Alphabet
      Marker
      Plain
    } derive(Eq)

    How a token's spelling turns back into text.

    Decoding::equal

    source
    fn Decoding::equal(Decoding, Decoding) -> Bool

    Decoding::not_equal

    source
    fn Decoding::not_equal(x : Decoding, y : Decoding) -> Bool

    Options

    source
    pub(all) struct Options {
      cache : Bool
      cache_limit : Int
    }

    Settings that change the work rather than the result.

    WordBpe

    source
    pub struct WordBpe {
      vocab : Map[String, Int]
      text_of : Map[Int, String]
      chars : Map[Char, Int]
      merges : Map[Int, Map[Int, Int]]
      results : Array[Int]
      unk_id : Int
      byte_ids : Array[Int]
      byte_fallback : Bool
      fuse_unk : Bool
      ignore_merges : Bool
      decoding : Decoding
      normalizer : 
    Normalizer
      split : 
    Split
      alphabet : Array[Char]
      reverse : Map[Int, Int]
    }

    A model that merges neighbouring tokens in the order a list of rules gives.

    This is the model a tokenizer.json file describes: a vocabulary of spellings, a list of merges, and the order they were learned in. It differs from a byte-level table in that the order is a list rather than a rank, and in that a character the vocabulary lacks is a hole rather than an impossibility.

    WordBpe::decode

    source
    fn WordBpe::decode(self : WordBpe, ids : ArrayView[Int]) -> String raise 
    Refused

    The text a list of tokens stands for.

    WordBpe::decode_bytes

    source
    fn WordBpe::decode_bytes(self : WordBpe, ids : ArrayView[Int]) -> Bytes raise 
    Refused

    The bytes a list of tokens stands for.

    WordBpe::encode

    source
    fn WordBpe::encode(self : WordBpe, text : StringView) -> Array[Int] raise 
    Refused

    The tokens of text.

    WordBpe::encode_word

    source
    fn WordBpe::encode_word(self : WordBpe, word : StringView) -> Array[Int] raise 
    Refused

    The tokens of one word, without cutting it first.

    WordBpe::length

    source
    fn WordBpe::length(self : WordBpe) -> Int

    How many tokens the vocabulary has.

    WordBpe::new

    source
    fn WordBpe::new(vocab~ : Array[(String, Int)], merges~ : Array[(String, String)], options? : WordOptions, decoding? : Decoding, normalizer? : 
    Normalizer    , split? : 
    Split    ) -> WordBpe raise 
    Refused

    Builds a model from a vocabulary and the merges between its tokens.

    merges are pairs of spellings in the order the trainer learned them; a merge whose sides are not both in the vocabulary is refused, because the result would be a rule that can never fire.

    WordBpe::size

    source
    fn WordBpe::size(self : WordBpe) -> Int

    The tokens of text, cut by the pre-tokenizer the file named.

    WordBpe::token_bytes

    source
    fn WordBpe::token_bytes(self : WordBpe, id : Int) -> Bytes?

    The spelling of one token.

    WordOptions

    source
    pub(all) struct WordOptions {
      unk_id : Int
      byte_fallback : Bool
      ignore_merges : Bool
    }

    Settings of a word-level byte-pair model.

    byte_alphabet

    source
    fn byte_alphabet() -> Array[Char]

    The alphabet the byte-level version of this model writes its vocabulary in: one character per byte, indexed by the byte.

    The bytes that are printable in Latin-1 keep their own character and the rest are laid out over the code points above U+0100, so that a token never holds a control character or half of a character.

    options

    source
    let options : Options

    word_options

    source
    let word_options : WordOptions

    Source Files

    Powered by MoonBit

    Site sourceReport issueModulesSkillsCLIBuild queueStatistics

    © 2026 mooncakes.io