🥮 mooncakes.io

    moonbitstack/moontoken/special does not have a README file

    Hit

    source
    pub(all) struct Hit {
      start : Int
      text : String
      id : Int
    }

    Where a special token was found.

    Policy

    source
    pub(all) enum Policy {
      Refuse
      Allow
      Ignore
      Only(Array[String])
    }

    What may happen to the tokens a model reserves.

    A model reserves a handful of spellings, <|endoftext|> and the like, that stand for structure rather than text. Whether text that happens to contain one is taken as the token, taken as text, or refused is the caller's decision, never the library's.

    Policy::allows

    source
    fn Policy::allows(self : Policy, text : StringView) -> Bool

    Whether policy recognises text.

    Policy::recognizes

    source
    fn Policy::recognizes(self : Policy) -> Bool

    Whether the policy looks for reserved tokens at all.

    Table

    source
    pub struct Table {
      texts : Array[String]
      ids : Array[Int]
      by_first : Map[Int, Array[Int]]
      index_of : Map[String, Int]
      text_of : Map[Int, String]
    }

    The tokens a model reserves, by spelling.

    Table::add

    source
    fn Table::add(self : Table, text : String, id : Int) -> Unit

    Adds a spelling. An empty spelling is ignored: it would match everywhere.

    Two spellings may share an id, which is how a vocabulary that renamed one of its placeholders keeps the old spelling readable; the first spelling added is the one the id is written back as, so the reverse direction stays single-valued.

    Table::empty

    source
    fn Table::empty() -> Table

    A table with nothing in it.

    Table::find

    source
    fn Table::find(self : Table, text : StringView, from : Int) -> Hit?

    The longest reserved token at or after from, and where it stands.

    Table::id

    source
    fn Table::id(self : Table, text : StringView) -> Int?

    The id of a spelling.

    Table::length

    source
    fn Table::length(self : Table) -> Int

    How many tokens the table holds.

    Table::of

    source
    fn Table::of(pairs : Array[(String, Int)]) -> Table

    A table from spellings and their ids.

    A spelling that appears twice keeps the first id, which is how a model file that lists the same control token twice still has one answer.

    Table::text

    source
    fn Table::text(self : Table, id : Int) -> String?

    The spelling of an id, when the id is one the table holds.

    cl100k_base

    source
    fn cl100k_base() -> Table

    The special tokens of cl100k_base, and of o200k_base and the two earlier encodings, which share the first one.

    The ids are the ones the published encodings give them, and the file of mergeable ranks does not carry them: they are part of the encoding's definition, not of the table a caller loads.

    o200k_base

    source
    fn o200k_base() -> Table

    The special tokens of o200k_base.

    o200k_harmony

    source
    fn o200k_harmony() -> Table

    The special tokens of o200k_harmony, the encoding the harmony chat format writes its control words with over the o200k_base table.

    Ten spellings carry meaning and the rest of the range the format reserves is spelled <|reserved_NNNNNN|>: a message written before a control word had a name still decodes, rather than the id falling off the end of the vocabulary. Id 200018 is both — <|endofprompt|> was named first, and the placeholder over the same id is kept as an alias, which is the one place a spelling and its id do not pair up.

    p50k_edit

    source
    fn p50k_edit() -> Table

    The special tokens of p50k_edit, the pair of r50k_base plus the three fill-in-the-middle markers.

    r50k_base

    source
    fn r50k_base() -> Table

    The special token of r50k_base and p50k_base.

    Source Files