🥮 mooncakes.io

    moonucd

    The Unicode Character Database for MoonBit.

    unicode
    ucd
    unicode-character-database
    normalization
    nfc
    nfd
    nfkc
    nfkd
    segmentation
    grapheme
    word-break
    sentence-break
    line-break
    uax14
    uax15
    uax29
    east-asian-width
    terminal-width
    wcwidth
    character-name
    unicode-name
    character-category
    binary-property
    bidi
    case-mapping
    case-folding
    uppercase
    lowercase
    numeric-value
    script
    block
    age
    uax44
    emoji
    moonbit
    Download zip
    Version
    0.2.0
    License
    Apache-2.0
    Repository
    Last updated
    11 hours ago
    Downloads
    7K

    moonucd

    moon add moonbitstack/moonucd

    assert_eq(@moonucd.category('Dž'), @moonucd.Lt)
    assert_eq(@moonucd.upper("straße"), "STRASSE")
    assert_true(@moonucd.fold("Straße") == @moonucd.fold("STRASSE"))
    assert_eq(@name.name('中'), Some("CJK UNIFIED IDEOGRAPH-4E2D"))
    assert_eq(@width.text_width("Hello 世界"), 10)
    assert_eq(@segment.grapheme_clusters("🇨🇳e\u{301}!"), ["🇨🇳", "e\u{301}", "!"])
    assert_eq(@linebreak.linebreaks("The quick 中文 fox"), [
      "The ", "quick ", "中", "文 ", "fox",
    ])

    #Packages

    The root package holds the general character facts. The other eight carry their own tables, so a program links only the tables of the packages it imports.

    Every function is pure and runs on native, wasm, wasm-gc and js. Offsets into strings are UTF-16 code units, the index MoonBit strings use.

    #A few more

    assert_true(@moonucd.has('_', XidContinue))
    assert_eq(@moonucd.lower("ΣΟΦΟΣ ΣΟΦΟΣ"), "σοφος σοφος")
    assert_eq(@moonucd.numeric_value('½'), Some(0.5))
    assert_eq(@moonucd.numeric_value('五'), Some(5))
    assert_eq(@moonucd.simple_upper('ß'), 'ß')
    assert_eq(@moonucd.simple_title('dž'), 'Dž')
    assert_eq(@age.age('€'), Some((2, 1)))
    assert_eq(@block.range("Greek"), Some(('\u{370}', '\u{3FF}')))
    assert_eq(@name.lookup("copyright sign"), Some('©'))
    assert_eq(@script.extensions('ー'), [@script.Hiragana, @script.Katakana])
    assert_eq(@normalize.normalize(NFKC, "ｈｅｌｌｏ"), "hello")
    assert_eq(@normalize.decomposition('½'), "<fraction> 0031 2044 0032")
    assert_eq(@width.width('α', ambiguous=2), 2)
    assert_eq(@segment.word_at("hello, world", 9), (7, 12))
    assert_eq(@linebreak.breaks("one two\nthree"), [(4, false), (8, true)])

    The guide walks through every package; the API reference lists every function and type.

    #Ranges and simple folding

    A set of characters is easier to work with as intervals than as a predicate: a pattern engine, a tokenizer or a character map all keep (start, end) pairs. Every category, every binary property and every script answers with its ranges — closed, ascending, neighbours already joined, so two sets meet by walking them once.

    assert_eq(@moonucd.Nd.ranges()[0], (0x30, 0x39))
    assert_eq(@moonucd.Dash.ranges().length(), 24)
    assert_eq(@moonucd.simple_fold('ς'), 'σ')
    assert_eq(@moonucd.fold_orbit('K'), ['K', 'k', '\u{212A}'])
    assert_eq(@script.Greek.ranges()[0], (0x370, 0x373))
    assert_eq(@script.Katakana.extended_ranges().length(), 22)

    simple_fold is the C and S classes of CaseFolding.txt: one character in, one character out, which is what a match that ignores case character by character needs. fold_orbit gives the whole class a character folds with, so a set closes under case without a walk of the code space. Script::ranges follows the script a character is listed under; Script::extended_ranges follows Script_Extensions, the scripts it is shared with.

    simple_upper, simple_lower and simple_title are the same one-character-in, one-character-out shape for case: a keyboard, a character map or a matcher that rewrites a character in place needs ß to stay ß, where upper writes SS. numeric_value answers the number a character counts as — 7, 0.5, 100 — and the CJK numerals take theirs from Unihan, where the UCD keeps them.

    #Examples

    ExampleShows
    01-tourcategory, numeric type, binary properties, case, ranges
    02-scriptsscript, extensions, Script::of
    03-widthwidth, east_asian_width, text_width
    04-normalizenormalize, is_normalized, decomposition
    05-segmentgrapheme_clusters, words, sentences
    06-linebreaklinebreaks, lb_property
    07-terminalwrapping and truncating text by terminal cells

    moon run src/examples/07-terminal

    #Conformance

    The algorithms are replayed against the test files the Unicode Standard ships with them: every case of each file, on every build.

    FileCases
    GraphemeBreakTest.txt766
    SentenceBreakTest.txt512
    WordBreakTest.txt1,944
    LineBreakTest.txt19,338
    NormalizationTest.txt20,034

    The properties with no test file of their own — the block, the script, the numeric values, the case forms — are compared code point by code point with two independent implementations of the same Unicode version, all 1,112,064 of them. Where an implementation and the Standard disagree the Standard wins: Tangut and the newest CJK extensions take their names from the Standard's rule NR2, and are named here even though one of the two implementations has no name for them.

    #Tables

    Every table is generated from the Unicode Character Database by the scripts in scripts/tables. Every package reports the version in version(), and a new Unicode version is a regeneration that leaves the interfaces as they are.

    #Licence

    Apache-2.0.

    Bidi

    source
    pub(all) enum Bidi {
      L
      R
      AL
      EN
      ES
      ET
      AN
      CS
      NSM
      BN
      B
      S
      WS
      ON
      LRE
      LRO
      RLE
      RLO
      PDF
      LRI
      RLI
      FSI
      PDI
    } derive(Eq, 
    Debug    )

    Bidi_Class.
    impl Show for Bidi

    Bidi::equal

    source
    fn Bidi::equal(Bidi, Bidi) -> Bool

    Bidi::not_equal

    source
    fn Bidi::not_equal(x : Bidi, y : Bidi) -> Bool

    Bidi::output

    source
    fn Bidi::output(self : Bidi, logger : &Logger) -> Unit

    Bidi::to_repr

    source

    Bidi::to_string

    source
    fn Bidi::to_string(self : Bidi) -> String

    Category

    source
    pub(all) enum Category {
      Lu
      Ll
      Lt
      Lm
      Lo
      Mn
      Mc
      Me
      Nd
      Nl
      No
      Pc
      Pd
      Ps
      Pe
      Pi
      Pf
      Po
      Sm
      Sc
      Sk
      So
      Zs
      Zl
      Zp
      Cc
      Cf
      Cs
      Co
      Cn
    } derive(Eq, 
    Debug    )

    General_Category.
    impl Show for Category

    Category::equal

    source
    fn Category::equal(Category, Category) -> Bool

    Category::not_equal

    source
    fn Category::not_equal(x : Category, y : Category) -> Bool

    Category::output

    source
    fn Category::output(self : Category, logger : &Logger) -> Unit

    Category::ranges

    source
    fn Category::ranges(self : Category) -> Array[(Int, Int)]

    The ranges the category covers: closed intervals, ascending by their start, neighbours joined, so that a matcher can hold a set of characters as a list instead of a predicate.

    test {
      let digits = @moonucd.Nd.ranges()
      assert_eq(digits[0], (0x30, 0x39))
      assert_eq(digits[1], (0x660, 0x669))
    }

    Category::to_repr

    source

    Category::to_string

    source
    fn Category::to_string(self : Category) -> String

    Numeric

    source
    pub(all) enum Numeric {
      Decimal
      Digit
      Numeric
    } derive(Eq, 
    Debug    )

    Numeric_Type, for characters that have one.

    Numeric::equal

    source
    fn Numeric::equal(Numeric, Numeric) -> Bool

    Numeric::not_equal

    source
    fn Numeric::not_equal(x : Numeric, y : Numeric) -> Bool

    Numeric::to_repr

    source

    Property

    source
    pub(all) enum Property {
      Alphabetic
      Lowercase
      Uppercase
      Cased
      CaseIgnorable
      ChangesWhenLowercased
      ChangesWhenUppercased
      ChangesWhenTitlecased
      ChangesWhenCasefolded
      ChangesWhenCasemapped
      IdStart
      IdContinue
      XidStart
      XidContinue
      DefaultIgnorableCodePoint
      GraphemeExtend
      GraphemeBase
      Math
      WhiteSpace
      BidiControl
      JoinControl
      Dash
      Hyphen
      QuotationMark
      TerminalPunctuation
      HexDigit
      AsciiHexDigit
      Ideographic
      Diacritic
      Extender
      NoncharacterCodePoint
      IdsBinaryOperator
      IdsTrinaryOperator
      Radical
      UnifiedIdeograph
      Deprecated
      SoftDotted
      LogicalOrderException
      SentenceTerminal
      VariationSelector
      PatternWhiteSpace
      PatternSyntax
      PrependedConcatenationMark
      RegionalIndicator
      Emoji
      EmojiPresentation
      EmojiModifier
      EmojiModifierBase
      EmojiComponent
      ExtendedPictographic
    } derive(Eq, 
    Debug    )

    A binary property from PropList, DerivedCoreProperties and emoji-data.

    Property::equal

    source
    fn Property::equal(Property, Property) -> Bool

    Property::not_equal

    source
    fn Property::not_equal(x : Property, y : Property) -> Bool

    Property::ranges

    source
    fn Property::ranges(self : Property) -> Array[(Int, Int)]

    The ranges the binary property covers: closed intervals, ascending by their start, neighbours joined.

    test {
      let dashes = @moonucd.Dash.ranges()
      assert_eq(dashes[0], (0x2D, 0x2D))
      assert_true(dashes.contains((0x2010, 0x2015)))
    }

    Property::to_repr

    source

    bidi

    source
    fn bidi(ch : Char) -> Bidi

    The bidirectional class of ch, with the defaults the UCD assigns to unassigned code points.

    category

    source
    fn category(ch : Char) -> Category

    The category of ch; unassigned code points are Cn.

    fold

    source
    fn fold(text : StringView) -> String

    text case-folded with the full foldings, for caseless comparison.

    fold_orbit

    source
    fn fold_orbit(ch : Char) -> Array[Char]

    Every character that folds to or from ch, itself included, ascending: the class a caseless match of one character has to cover.

    test {
      assert_eq(@moonucd.fold_orbit('K'), ['K', 'k', '\u{212A}'])
      assert_eq(@moonucd.fold_orbit('ς'), ['Σ', 'ς', 'σ'])
      assert_eq(@moonucd.fold_orbit('中'), ['中'])
    }

    has

    source
    fn has(ch : Char, property : Property) -> Bool

    Whether ch has property.

    lower

    source
    fn lower(text : StringView) -> String

    text in lowercase, with the full mappings; a capital sigma at the end of a word becomes ς.

    numeric

    source
    fn numeric(ch : Char) -> Numeric?

    The numeric type of ch; None for characters without a numeric value.

    numeric_value

    source
    fn numeric_value(ch : Char) -> Double?

    The numeric value of ch: 7 for '7', 0.5 for '½', 100 for '佰'; None for characters that count nothing.

    test {
      assert_eq(@moonucd.numeric_value('7'), Some(7.0))
      assert_eq(@moonucd.numeric_value('٣'), Some(3.0))
      assert_eq(@moonucd.numeric_value('½'), Some(0.5))
      assert_eq(@moonucd.numeric_value('五'), Some(5.0))
      assert_eq(@moonucd.numeric_value('中'), None)
    }

    simple_fold

    source
    fn simple_fold(ch : Char) -> Char

    The simple case folding of ch: the one character it turns into when case stops counting, from the C and S classes of CaseFolding.txt. A code point that only has a longer fold, ß to ss, folds to itself, since one character cannot become two.

    test {
      assert_eq(@moonucd.simple_fold('A'), 'a')
      assert_eq(@moonucd.simple_fold('ς'), 'σ')
      assert_eq(@moonucd.simple_fold('ß'), 'ß')
      assert_eq(@moonucd.simple_fold('1'), '1')
    }

    simple_lower

    source
    fn simple_lower(ch : Char) -> Char

    The simple lowercase mapping of ch: one character in, one character out. İ becomes i here, where lower writes i and a combining dot.

    test {
      assert_eq(@moonucd.simple_lower('A'), 'a')
      assert_eq(@moonucd.simple_lower('İ'), 'i')
      assert_eq(@moonucd.simple_lower('Σ'), 'σ')
    }

    simple_title

    source
    fn simple_title(ch : Char) -> Char

    The simple titlecase mapping of ch: one character in, one character out. A character with no titlecase mapping of its own takes its uppercase one, which is why dž becomes Dž or DŽ depending on which call is used.

    test {
      assert_eq(@moonucd.simple_title('dž'), 'Dž')
      assert_eq(@moonucd.simple_title('a'), 'A')
      assert_eq(@moonucd.simple_title('中'), '中')
    }

    simple_upper

    source
    fn simple_upper(ch : Char) -> Char

    The simple uppercase mapping of ch: one character in, one character out, as UnicodeData lists it. ß stays ß, where upper writes SS; a caller that maps a character in place, a matcher or a keyboard, wants this one.

    test {
      assert_eq(@moonucd.simple_upper('a'), 'A')
      assert_eq(@moonucd.simple_upper('ß'), 'ß')
      assert_eq(@moonucd.simple_upper('Dž'), 'DŽ')
      assert_eq(@moonucd.simple_upper('中'), '中')
    }

    title

    source
    fn title(text : StringView) -> String

    Every character of text mapped to its titlecase form. Words are not found here: callers decide which characters start one.

    upper

    source
    fn upper(text : StringView) -> String

    text in uppercase, with the full mappings: ß becomes SS.

    version

    source
    fn version() -> String

    The version of the Unicode Character Database the tables come from.

    Source Files