README- URefused
- TrModel
- ::byte_bpeTokenizer::byte_bpe
- ::cl100kTokenizer::cl100k
- ::countTokenizer::count
- ::decodeTokenizer::decode
- ::decode_bytesTokenizer::decode_bytes
- ::encodeTokenizer::encode
- ::encode_ordinaryTokenizer::encode_ordinary
- ::lengthTokenizer::length
- ::newTokenizer::new
- ::o200kTokenizer::o200k
- ::ofTokenizer::of
- ::of_protoTokenizer::of_proto
- ::p50k_editTokenizer::p50k_edit
- ::parseTokenizer::parse
- ::piece_bpeTokenizer::piece_bpe
- ::policyTokenizer::policy
- ::r50kTokenizer::r50k
- ::token_bytesTokenizer::token_bytes
- ::unigramTokenizer::unigram
- ::with_policyTokenizer::with_policy
moonbitstack/moontoken/refused does not have a README file
Refused
pub(all) suberror Refused {
Malformed(String, Int)
Absent(String)
Unsupported(String)
Missing(Int)
Disallowed(String, Int)
}Why an input was refused.
Malformed carries the offset, Absent the name of what was missing, and
Missing the id that is not in the vocabulary, so that a caller reading a
table of a hundred thousand entries can find the one that broke.
Source Files
Install
Download zipTokenizers for MoonBit: byte-level BPE and SentencePiece.
tokenizertokenizerstokenizationtokencount-tokensbpebyte-pair-encodingbyte-level-bpesentencepieceunigramtiktokencl100kcl100k_baseo200ko200k_baser50k_basep50k_basetokenizer-jsonvocabularyvocabmergesencodedecodespecial-tokensbyte-fallbackpre-tokenizernormalizationnlpllmpromptcontext-windowmoonbit
Apache-2.0
Updated 13 hours ago
10 downloads