🥮 mooncakes.io

    moonbitstack/moontoken/vocab does not have a README file

    Ranks

    source
    pub struct Ranks {
      encoder : Map[Bytes, Int]
      decoder : Map[Int, Bytes]
    }

    The mergeable ranks of a byte-level encoding: byte sequences and the order in which the trainer picked them.

    A rank doubles as the token id, which is how the published encodings are numbered, and the reason a table loaded here needs no second index.

    Ranks::empty

    source
    fn Ranks::empty() -> Ranks

    An empty table.

    Ranks::from_pairs

    source
    fn Ranks::from_pairs(pairs : Array[(Bytes, Int)]) -> Ranks raise 
    Refused

    A table from tokens and their ranks.

    Raises when one token or one rank appears twice: a table like that has no single answer for an id, and the encodings never contain one.

    Ranks::from_tiktoken

    source
    fn Ranks::from_tiktoken(text : StringView) -> Ranks raise 
    Refused

    Reads the layout the published encodings ship: one base64 rank pair per line, in whatever order.

    Ranks::get

    source
    fn Ranks::get(self : Ranks, token : BytesView) -> Int?

    The rank of a token, or None when the table does not have it.

    Ranks::length

    source
    fn Ranks::length(self : Ranks) -> Int

    How many tokens the table holds.

    Ranks::missing_bytes

    source
    fn Ranks::missing_bytes(self : Ranks) -> Array[Int]

    The bytes that have no token of their own.

    A byte-level encoding is total over byte strings only when every byte is in the table; without this check an encoding built from a partial table would fail in the middle of a word instead of at the door.

    Ranks::token

    source
    fn Ranks::token(self : Ranks, rank : Int) -> Bytes?

    The token of a rank.

    Source Files

    Powered by MoonBit

    Site sourceReport issueModulesSkillsCLIBuild queueStatistics

    © 2026 mooncakes.io