moonbitstack/moontoken/split

README
U
Refused
Tr
Model
- ::byte_bpeTokenizer::byte_bpe
- ::cl100kTokenizer::cl100k
- ::countTokenizer::count
- ::decodeTokenizer::decode
- ::decode_bytesTokenizer::decode_bytes
- ::encodeTokenizer::encode
- ::encode_ordinaryTokenizer::encode_ordinary
- ::lengthTokenizer::length
- ::newTokenizer::new
- ::o200kTokenizer::o200k
- ::ofTokenizer::of
- ::of_protoTokenizer::of_proto
- ::p50k_editTokenizer::p50k_edit
- ::parseTokenizer::parse
- ::piece_bpeTokenizer::piece_bpe
- ::policyTokenizer::policy
- ::r50kTokenizer::r50k
- ::token_bytesTokenizer::token_bytes
- ::unigramTokenizer::unigram
- ::with_policyTokenizer::with_policy

moonbitstack/moontoken/split does not have a README file

Digits

pub(all) struct Digits {
  individual : Bool
}

Whether digits are cut one at a time or as a run.

Split

source

pub(all) enum Split {
  Whole
  Gpt2
  Cl100k
  O200k
  Whitespace
  Metaspace
  Digits(Digits)
  Sequence(Array[Split])
}

How text is cut into the words a model is applied to.

The three named rules are the published pre-tokenizers of the byte-level encodings: r50k/p50k/gpt2, cl100k_base and o200k_base. They are scanners here rather than regular expressions because the patterns lean on lookahead and possessive quantifiers, and because a tokenizer runs them once per word over the whole input.

Split::pieces

source

fn Split::pieces(self : Split, text : StringView) -> Array[StringView]

The words of text, in order. Joining them gives back the text under every rule but Whitespace, which drops what it splits on.

digits_individual

source

let digits_individual : Digits

digits_run

source

let digits_run : Digits

utf8_width_at

source

fn utf8_width_at(bytes : Bytes, at : Int) -> Int

The width of the UTF-8 sequence that starts at at, or 0 when the bytes there are not one.

Source Files

Install

Download zip

Tokenizers for MoonBit: byte-level BPE and SentencePiece.

tokenizertokenizerstokenizationtokencount-tokensbpebyte-pair-encodingbyte-level-bpesentencepieceunigramtiktokencl100kcl100k_baseo200ko200k_baser50k_basep50k_basetokenizer-jsonvocabularyvocabmergesencodedecodespecial-tokensbyte-fallbackpre-tokenizernormalizationnlpllmpromptcontext-windowmoonbit

moonbitstack

Apache-2.0