🥮 mooncakes.io

    moonbitstack/moontoken/normalizer does not have a README file

    Mode

    source
    pub enum Mode {
      Identity
      Rule
      Chain
    }

    Which of the three normalisations this is.

    Normalizer

    source
    pub struct Normalizer {
      mode : Mode
      add_dummy_prefix : Bool
      remove_extra_whitespaces : Bool
      escape_whitespaces : Bool
      treat_whitespace_as_suffix : Bool
      trie : Array[UInt]
      replacements : Bytes
      user_defined : Map[Int, Array[Bytes]]
      stages : Array[Stage]
    }

    The normalisation applied before segmentation.

    A model file carries its own rule table, which is why this is not just a list of operations: the table is a trie of a few thousand replacements, and applying it is what makes two spellings of one character segment alike.

    Normalizer::apply

    source
    fn Normalizer::apply(self : Normalizer, text : StringView) -> String raise 
    Refused

    The text as it goes into segmentation.

    Normalizer::chain

    source
    fn Normalizer::chain(stages : Array[Stage]) -> Normalizer

    A normalisation from a list of operations, applied in order.

    Normalizer::identity

    source
    fn Normalizer::identity() -> Normalizer

    A normalisation that changes nothing.

    Normalizer::rule

    source
    fn Normalizer::rule(spec~ : Spec, user_defined? : Array[Bytes], treat_whitespace_as_suffix? : Bool) -> Normalizer raise 
    Refused

    The rule table of a model file.

    user_defined are the symbols the caller reserved: the normaliser hands them through untouched, so that a piece spelled the way the caller wrote it stays findable.

    Normalizer::strips_every_leading_marker

    source
    fn Normalizer::strips_every_leading_marker(self : Normalizer) -> Bool

    Whether every marker at the front comes off, not only the first.

    Normalizer::strips_leading_marker

    source
    fn Normalizer::strips_leading_marker(self : Normalizer) -> Bool

    Whether a word-break marker at the very front of the text stands for a space the text does not have, rather than one it does.

    A model that adds a leading space puts a marker there on the way in, and taking it off again on the way out is what makes the round trip exact. A model that also drops runs of whitespace keeps taking them off until the text has a character in it, because everything it saw at the front was whitespace it meant to discard.

    Spec

    source
    pub(all) struct Spec {
      name : String
      charsmap : Bytes
      add_dummy_prefix : Bool
      remove_extra_whitespaces : Bool
      escape_whitespaces : Bool
    }

    The normalisation a model expects: the rule table it was trained with, and the three switches around whitespace.

    Stage

    source
    pub(all) enum Stage {
      Nfc
      Nfd
      Nfkc
      Nfkd
      Lowercase
      Prepend(String)
      Pad(String)
      Replace(String, String)
      Trim
      EscapeSpace
    }

    One step of a normalisation that is a list of Unicode operations rather than a rule table.

    longest_reserved

    source
    fn longest_reserved(index : Map[Int, Array[Bytes]], input : Bytes, at : Int) -> Int?

    The number of bytes of the longest reserved symbol that starts at at.

    user_index

    source
    fn user_index(symbols : Array[Bytes]) -> Map[Int, Array[Bytes]]

    An index of the symbols a caller reserved, by their first byte, so that finding the one at a position costs a map lookup rather than a scan of the whole list.

    Source Files

    Powered by MoonBit

    Site sourceReport issueModulesSkillsCLIBuild queueStatistics

    © 2026 mooncakes.io