🥮 mooncakes.io

    moonbitstack/moontoken/special does not have a README file

    Hit

    source
    pub(all) struct Hit {
      start : Int
      text : String
      id : Int
    }

    Where a special token was found.

    Policy

    source
    pub(all) enum Policy {
      Refuse
      Allow
      Ignore
      Only(Array[String])
    }

    What may happen to the tokens a model reserves.

    A model reserves a handful of spellings, <|endoftext|> and the like, that stand for structure rather than text. Whether text that happens to contain one is taken as the token, taken as text, or refused is the caller's decision, never the library's.

    Policy::allows

    source
    fn Policy::allows(self : Policy, text : StringView) -> Bool

    Whether policy recognises text.

    Policy::recognizes

    source
    fn Policy::recognizes(self : Policy) -> Bool

    Whether the policy looks for reserved tokens at all.

    Table

    source
    pub struct Table {
      texts : Array[String]
      ids : Array[Int]
      by_first : Map[Int, Array[Int]]
      index_of : Map[String, Int]
      text_of : Map[Int, String]
    }

    The tokens a model reserves, by spelling.

    Table::add

    source
    fn Table::add(self : Table, text : String, id : Int) -> Unit

    Adds a spelling. An empty spelling is ignored: it would match everywhere.

    Table::empty

    source
    fn Table::empty() -> Table

    A table with nothing in it.

    Table::find

    source
    fn Table::find(self : Table, text : StringView, from : Int) -> Hit?

    The longest reserved token at or after from, and where it stands.

    Table::id

    source
    fn Table::id(self : Table, text : StringView) -> Int?

    The id of a spelling.

    Table::length

    source
    fn Table::length(self : Table) -> Int

    How many tokens the table holds.

    Table::of

    source
    fn Table::of(pairs : Array[(String, Int)]) -> Table

    A table from spellings and their ids.

    A spelling that appears twice keeps the first id, which is how a model file that lists the same control token twice still has one answer.

    Table::text

    source
    fn Table::text(self : Table, id : Int) -> String?

    The spelling of an id, when the id is one the table holds.

    cl100k_base

    source
    fn cl100k_base() -> Table

    The special tokens of cl100k_base, and of o200k_base and the two earlier encodings, which share the first one.

    The ids are the ones the published encodings give them, and the file of mergeable ranks does not carry them: they are part of the encoding's definition, not of the table a caller loads.

    o200k_base

    source
    fn o200k_base() -> Table

    The special tokens of o200k_base.

    p50k_edit

    source
    fn p50k_edit() -> Table

    The special tokens of p50k_edit, the pair of r50k_base plus the three fill-in-the-middle markers.

    r50k_base

    source
    fn r50k_base() -> Table

    The special token of r50k_base and p50k_base.

    Source Files

    Powered by MoonBit

    Site sourceReport issueModulesSkillsCLIBuild queueStatistics

    © 2026 mooncakes.io