🥮 mooncakes.io

    unicode-data

    Unicode Character Database lookup and case mapping for MoonBit

    unicode
    ucd
    case-mapping
    general-category
    Download zip
    Version
    0.7.1
    License
    Apache-2.0
    Repository
    Last updated
    8 days ago
    Downloads
    23K

    #unicode-data

    Unicode Character Database lookup, general categories, East Asian width, line break classes, joining types and groups, Hangul syllable types, binary properties (White_Space, Alphabetic, Lowercase, Uppercase, XID_Start, XID_Continue, Default_Ignorable_Code_Point, Grapheme_Extend, Regional_Indicator, Prepended_Concatenation_Mark, Emoji_Presentation, Emoji_Modifier, Emoji_Modifier_Base, Extended_Pictographic), emoji variation sequences, and case mapping for MoonBit. This module uses Unicode 18.0.0 data.

    moon add moonbit-community/unicode-data

    See the repository root README for API documentation and examples.

    #Low-level data packages

    Import the package for the lookup you need. Each package owns its generated tables and exposes lookup functions; the tables themselves remain private.

    Package under moonbit-community/unicode-data/Lookups
    decompositionCanonical and compatibility decomposition mappings
    compositionComposition pairs and composition exclusions
    cccCanonical Combining Class
    general_categoryGeneral Category ordinal and is_mark
    punctuationGeneral_Category group P (punctuation)
    east_asian_widthEast_Asian_Width ordinal
    case_mappingSimple and full uppercase, lowercase, and titlecase mappings
    xidXID_Start and XID_Continue
    alphabeticAlphabetic
    lowercaseLowercase
    uppercaseUppercase
    white_spaceWhite_Space
    default_ignorable_code_pointDefault_Ignorable_Code_Point
    grapheme_extendGrapheme_Extend
    regional_indicatorRegional_Indicator
    prepended_concatenation_markPrepended_Concatenation_Mark
    emojiExtended_Pictographic, Emoji_Presentation, Emoji_Modifier, Emoji_Modifier_Base, and the characters starting an emoji variation sequence
    joining_typeJoining_Type ordinal
    joining_groupJoining_Group ordinal
    hangul_syllable_typeHangul_Syllable_Type ordinal
    line_breakLine_Break ordinal

    moonbit-community/unicode-data/data remains a compatibility package that re-exports the old lookup functions with deprecation notices pointing to these packages. The high-level moonbit-community/unicode-data interface is unchanged: its general_category(c) returns GeneralCategory, with GeneralCategoryGroup available through .group(). These enums remain defined in the root package. The low-level general_category package retains lookup_general_category(c) -> Int, returning the existing 0–29 category ordinals.

    EastAsianWidth

    source
    pub(all) enum EastAsianWidth {
      A
      F
      H
      N
      Na
      W
    } derive(Eq, 
    Debug    )

    Unicode East_Asian_Width values See UAX #11: https://www.unicode.org/reports/tr11/

    EastAsianWidth::equal

    source

    EastAsianWidth::not_equal

    source
    fn EastAsianWidth::not_equal(x : EastAsianWidth, y : EastAsianWidth) -> Bool

    EastAsianWidth::output

    source
    fn EastAsianWidth::output(self : EastAsianWidth, logger : &Logger) -> Unit

    EastAsianWidth::to_string

    source
    fn EastAsianWidth::to_string(self : EastAsianWidth) -> String

    GeneralCategory

    source
    pub(all) enum GeneralCategory {
      Lu
      Ll
      Lt
      Lm
      Lo
      Mn
      Mc
      Me
      Nd
      Nl
      No
      Zs
      Zl
      Zp
      Pc
      Pd
      Ps
      Pe
      Pi
      Pf
      Po
      Sm
      Sc
      Sk
      So
      Cc
      Cf
      Cs
      Co
      Cn
    } derive(Eq, 
    Debug    )

    Unicode General_Category values (two-letter categories) See UAX #44: https://www.unicode.org/reports/tr44/#General_Category_Values

    GeneralCategory::equal

    source

    GeneralCategory::group

    source

    Get the major category group (single-letter category) for a GeneralCategory

    GeneralCategory::not_equal

    source
    fn GeneralCategory::not_equal(x : GeneralCategory, y : GeneralCategory) -> Bool

    GeneralCategory::output

    source
    fn GeneralCategory::output(self : GeneralCategory, logger : &Logger) -> Unit

    GeneralCategory::to_string

    source
    fn GeneralCategory::to_string(self : GeneralCategory) -> String

    GeneralCategoryGroup

    source
    pub(all) enum GeneralCategoryGroup {
      L
      M
      N
      Z
      P
      S
      C
    } derive(Eq, 
    Debug    )

    Unicode General_Category major class (single-letter categories) See UAX #44: https://www.unicode.org/reports/tr44/#General_Category_Values

    GeneralCategoryGroup::equal

    source

    GeneralCategoryGroup::not_equal

    source

    GeneralCategoryGroup::output

    source
    fn GeneralCategoryGroup::output(self : GeneralCategoryGroup, logger : &Logger) -> Unit

    GeneralCategoryGroup::to_string

    source
    fn GeneralCategoryGroup::to_string(self : GeneralCategoryGroup) -> String

    HangulSyllableType

    source
    pub(all) enum HangulSyllableType {
      NotApplicable
      LeadingJamo
      VowelJamo
      TrailingJamo
      LVSyllable
      LVTSyllable
    } derive(Eq, 
    Debug    )

    Unicode Hangul_Syllable_Type values See the Unicode Standard, section 3.12, Conjoining Jamo Behavior: https://www.unicode.org/versions/latest/core-spec/chapter-3/#G24646

    HangulSyllableType::equal

    source

    HangulSyllableType::not_equal

    source

    HangulSyllableType::output

    source
    fn HangulSyllableType::output(self : HangulSyllableType, logger : &Logger) -> Unit

    HangulSyllableType::to_string

    source
    fn HangulSyllableType::to_string(self : HangulSyllableType) -> String

    JoiningGroup

    source
    pub(all) enum JoiningGroup {
      AfricanFeh
      AfricanNoon
      AfricanQaf
      Ain
      Alaph
      Alef
      Beh
      Beth
      BurushaskiYehBarree
      CrownAin
      CrownBeh
      CrownFeh
      CrownHah
      CrownHeh
      CrownKaf
      CrownMeem
      CrownSad
      CrownSeen
      CrownTah
      Dal
      DalathRish
      E
      FarsiYeh
      Fe
      Feh
      FinalSemkath
      Gaf
      Gamal
      Hah
      HanifiRohingyaKinnaYa
      HanifiRohingyaPa
      He
      Heh
      HehGoal
      Heth
      Kaf
      Kaph
      KashmiriYeh
      Khaph
      KnottedHeh
      Lam
      Lamadh
      MalayalamBha
      MalayalamJa
      MalayalamLla
      MalayalamLlla
      MalayalamNga
      MalayalamNna
      MalayalamNnna
      MalayalamNya
      MalayalamRa
      MalayalamSsa
      MalayalamTta
      ManichaeanAleph
      ManichaeanAyin
      ManichaeanBeth
      ManichaeanDaleth
      ManichaeanDhamedh
      ManichaeanFive
      ManichaeanGimel
      ManichaeanHeth
      ManichaeanHundred
      ManichaeanKaph
      ManichaeanLamedh
      ManichaeanMem
      ManichaeanNun
      ManichaeanOne
      ManichaeanPe
      ManichaeanQoph
      ManichaeanResh
      ManichaeanSadhe
      ManichaeanSamekh
      ManichaeanTaw
      ManichaeanTen
      ManichaeanTeth
      ManichaeanThamedh
      ManichaeanTwenty
      ManichaeanWaw
      ManichaeanYodh
      ManichaeanZayin
      Meem
      Mim
      NoJoiningGroup
      Noon
      Nun
      Nya
      Pe
      Qaf
      Qaph
      Reh
      ReversedPe
      RohingyaYeh
      Sad
      Sadhe
      Seen
      Semkath
      Shin
      StraightWaw
      SwashKaf
      SyriacWaw
      Tah
      Taw
      TehMarbuta
      TehMarbutaGoal
      Teth
      ThinNoon
      ThinYeh
      VerticalTail
      Waw
      Yeh
      YehBarree
      YehWithTail
      Yudh
      YudhHe
      Zain
      Zhain
    } derive(Eq, 
    Debug    )

    Unicode Joining_Group values See the Unicode Standard, section 9.2, Arabic: https://www.unicode.org/versions/latest/core-spec/chapter-9/#G7480

    JoiningGroup::equal

    source

    JoiningGroup::not_equal

    source
    fn JoiningGroup::not_equal(x : JoiningGroup, y : JoiningGroup) -> Bool

    JoiningGroup::output

    source
    fn JoiningGroup::output(self : JoiningGroup, logger : &Logger) -> Unit

    JoiningGroup::to_string

    source
    fn JoiningGroup::to_string(self : JoiningGroup) -> String

    JoiningType

    source
    pub(all) enum JoiningType {
      NonJoining
      JoinCausing
      DualJoining
      LeftJoining
      RightJoining
      Transparent
    } derive(Eq, 
    Debug    )

    Unicode Joining_Type values See the Unicode Standard, section 9.2, Arabic: https://www.unicode.org/versions/latest/core-spec/chapter-9/#G7480
    impl Show for JoiningType

    JoiningType::equal

    source
    fn JoiningType::equal(JoiningType, JoiningType) -> Bool

    JoiningType::not_equal

    source
    fn JoiningType::not_equal(x : JoiningType, y : JoiningType) -> Bool

    JoiningType::output

    source
    fn JoiningType::output(self : JoiningType, logger : &Logger) -> Unit

    JoiningType::to_string

    source
    fn JoiningType::to_string(self : JoiningType) -> String

    LineBreak

    source
    pub(all) enum LineBreak {
      AI
      AK
      AL
      AP
      AS
      B2
      BA
      BB
      BK
      CB
      CJ
      CL
      CM
      CP
      CR
      EB
      EM
      EX
      GL
      H2
      H3
      HH
      HL
      HY
      ID
      IN
      IS
      JL
      JT
      JV
      LF
      NL
      NS
      NU
      OP
      PO
      PR
      QU
      RI
      SA
      SG
      SP
      SY
      VF
      VI
      WJ
      XX
      ZW
      ZWJ
    } derive(Eq, 
    Debug    )

    Unicode Line_Break values See UAX #14: https://www.unicode.org/reports/tr14/#Table1
    impl Show for LineBreak

    LineBreak::equal

    source
    fn LineBreak::equal(LineBreak, LineBreak) -> Bool

    LineBreak::not_equal

    source
    fn LineBreak::not_equal(x : LineBreak, y : LineBreak) -> Bool

    LineBreak::output

    source
    fn LineBreak::output(self : LineBreak, logger : &Logger) -> Unit

    LineBreak::to_string

    source
    fn LineBreak::to_string(self : LineBreak) -> String

    east_asian_width

    source
    fn east_asian_width(c : Char) -> EastAsianWidth

    Get the East_Asian_Width for a character.

    East_Asian_Width classifies characters by the width they have in East Asian typography. It is a property of a single character, not a column count: nonspacing marks and control characters have a value too. Code points the Unicode Character Database does not list are Neutral, except that unassigned code points in the CJK ideograph blocks and in planes 2 and 3 are Wide.

    general_category

    source
    fn general_category(c : Char) -> GeneralCategory

    Get the General_Category for a character

    hangul_syllable_type

    source
    fn hangul_syllable_type(c : Char) -> HangulSyllableType

    Get the Hangul_Syllable_Type for a character.

    Hangul_Syllable_Type tells the conjoining jamo apart from the precomposed syllables: a leading consonant, a vowel, a trailing consonant, a syllable of a leading consonant and a vowel, or one that also has a trailing consonant. Every other character, including the Hangul compatibility jamo and unassigned code points, is NotApplicable.

    It is narrower than Grapheme_Cluster_Break, whose V value also covers a few characters outside Hangul.

    has_emoji_variation_sequence

    source
    fn has_emoji_variation_sequence(c : Char) -> Bool

    Check whether a character starts an emoji variation sequence.

    These are the characters emoji-variation-sequences.txt lists: each can be followed by U+FE0E VARIATION SELECTOR-15 to ask for a text presentation or by U+FE0F VARIATION SELECTOR-16 to ask for an emoji presentation. The set contains characters on both sides of Emoji_Presentation, such as U+2764 HEAVY BLACK HEART, which is text by default, and U+231A WATCH, which is emoji by default. Most emoji are not in it, since they have no text presentation to select.

    is_alphabetic

    source
    fn is_alphabetic(c : Char) -> Bool

    Check whether a character has the Unicode Alphabetic property.

    Alphabetic covers the letters (General_Category L), letter numbers (Nl), and Other_Alphabetic characters such as the vowel signs of Indic scripts. Decimal digits are not Alphabetic.

    is_default_ignorable_code_point

    source
    fn is_default_ignorable_code_point(c : Char) -> Bool

    Check whether a character has the Unicode Default_Ignorable_Code_Point property.

    Default_Ignorable_Code_Point covers characters that should not be displayed when a font has no glyph for them, such as U+00AD SOFT HYPHEN, the zero width joiners, the bidirectional controls, the variation selectors, and the tag characters. It also covers some unassigned code points reserved for future characters of this kind.

    is_emoji_modifier

    source
    fn is_emoji_modifier(c : Char) -> Bool

    Check whether a character has the Unicode Emoji_Modifier property.

    Emoji_Modifier covers the five skin tone modifiers U+1F3FB to U+1F3FF.

    is_emoji_modifier_base

    source
    fn is_emoji_modifier_base(c : Char) -> Bool

    Check whether a character has the Unicode Emoji_Modifier_Base property.

    Emoji_Modifier_Base covers the characters whose appearance a following Emoji_Modifier changes, such as U+1F44D THUMBS UP SIGN.

    is_emoji_presentation

    source
    fn is_emoji_presentation(c : Char) -> Bool

    Check whether a character has the Unicode Emoji_Presentation property.

    Emoji_Presentation covers the characters displayed as emoji by default. It is narrower than the Emoji property: U+2764 HEAVY BLACK HEART and the ASCII digits are emoji only when followed by U+FE0F VARIATION SELECTOR-16. See UTS #51: https://www.unicode.org/reports/tr51/

    is_extended_pictographic

    source
    fn is_extended_pictographic(c : Char) -> Bool

    Check the Unicode Extended_Pictographic property, including reserved code points intended for future emoji. Used by UAX #14 and UAX #29.

    is_grapheme_extend

    source
    fn is_grapheme_extend(c : Char) -> Bool

    Check whether a character has the Unicode Grapheme_Extend property.

    Grapheme_Extend covers the nonspacing and enclosing marks (General_Category Mn and Me) and Other_Grapheme_Extend characters such as U+200C ZERO WIDTH NON-JOINER and a few spacing marks. U+200D ZERO WIDTH JOINER is not Grapheme_Extend.

    is_lowercase

    source
    fn is_lowercase(c : Char) -> Bool

    Check whether a character has the Unicode Lowercase property.

    Lowercase is broader than General_Category Ll: it also covers Other_Lowercase characters such as U+00AA FEMININE ORDINAL INDICATOR, modifier letters like U+02B0, and the small Roman numerals. Titlecase letters (Lt) are neither Lowercase nor Uppercase.

    is_prepended_concatenation_mark

    source
    fn is_prepended_concatenation_mark(c : Char) -> Bool

    Check whether a character has the Unicode Prepended_Concatenation_Mark property.

    Prepended_Concatenation_Mark covers format characters that enclose or extend under the digits or letters following them, such as U+0600 ARABIC NUMBER SIGN and U+06DD ARABIC END OF AYAH. Unlike most format characters they are visible.

    is_punctuation

    source
    fn is_punctuation(c : Char) -> Bool

    Check whether a character is punctuation.

    Punctuation is the General_Category group P, covering the connector, dash, open, close, initial, final, and other punctuation categories (Pc, Pd, Ps, Pe, Pi, Pf, and Po). Symbols such as $ (Sc) and + (Sm) are not punctuation.

    The predicate uses its own compact table instead of the General_Category trie, so code that only tests punctuation does not link the whole category table.

    is_regional_indicator

    source
    fn is_regional_indicator(c : Char) -> Bool

    Check whether a character has the Unicode Regional_Indicator property.

    Regional_Indicator covers the 26 regional indicator symbols U+1F1E6 to U+1F1FF, which pair up to form emoji flags.

    is_uppercase

    source
    fn is_uppercase(c : Char) -> Bool

    Check whether a character has the Unicode Uppercase property.

    Uppercase is broader than General_Category Lu: it also covers Other_Uppercase characters such as the capital Roman numerals and the circled capital Latin letters. Titlecase letters (Lt) are neither Lowercase nor Uppercase.

    is_white_space

    source
    fn is_white_space(c : Char) -> Bool

    Check whether a character has the Unicode White_Space property.

    White_Space covers the ASCII controls TAB, LF, VT, FF, and CR, the space separators (General_Category Zs), U+0085 NEXT LINE, and the line and paragraph separators U+2028 and U+2029. It does not include U+200B ZERO WIDTH SPACE or U+FEFF ZERO WIDTH NO-BREAK SPACE.

    is_xid_continue

    source
    fn is_xid_continue(c : Char) -> Bool

    Check whether a character has the Unicode XID_Continue property.

    XID_Continue is the recommended continuation class for Unicode default identifiers and includes every XID_Start character.

    is_xid_start

    source
    fn is_xid_start(c : Char) -> Bool

    Check whether a character has the Unicode XID_Start property.

    XID_Start is the recommended start class for Unicode default identifiers.

    joining_group

    source
    fn joining_group(c : Char) -> JoiningGroup

    Get the Joining_Group for a character.

    Joining_Group classifies the letters of cursive scripts such as Arabic, Syriac, and Manichaean by the shape they join with: letters that differ only in their dots, like U+0628 ARABIC LETTER BEH and U+062A ARABIC LETTER TEH, share a group. Characters that are not such letters, including unassigned code points, are NoJoiningGroup.

    The value Pe shares its name with a GeneralCategory value: write JoiningGroup::Pe or GeneralCategory::Pe where the type is not known from the context.

    joining_type

    source
    fn joining_type(c : Char) -> JoiningType

    Get the Joining_Type for a character.

    Joining_Type describes how a character takes part in cursive joining in scripts such as Arabic, Syriac, and Mongolian. Transparent characters, mostly nonspacing marks and format characters, do not interrupt joining between the characters on either side of them. Characters that take no part in cursive joining, including unassigned code points, are NonJoining.

    line_break

    source
    fn line_break(c : Char) -> LineBreak

    Get the Line_Break for a character.

    Line_Break is the property the line breaking rules of UAX #14 are written in terms of. It is a property of a single character, and the value is the one the Unicode Character Database assigns: rule LB1 of the algorithm, which resolves AI, CJ, SA, SG, and XX to other classes, is not applied. Unassigned code points are XX, except in ranges reserved for characters of a known class, such as ID in the CJK ideograph blocks and in planes 2 and 3.

    to_lowercase

    source
    fn to_lowercase(c : Char) -> String

    Get the full lowercase mapping for a character Returns a String containing one or more lowercase characters For example, 'İ' (U+0130) maps to "i\u{0307}" (i with combining dot above)

    to_simple_lowercase

    source
    fn to_simple_lowercase(c : Char) -> Char

    Get the simple lowercase mapping for a character Returns the character itself if no mapping exists

    to_simple_titlecase

    source
    fn to_simple_titlecase(c : Char) -> Char

    Get the simple titlecase mapping for a character Returns the character itself if no mapping exists

    to_simple_uppercase

    source
    fn to_simple_uppercase(c : Char) -> Char

    Get the simple uppercase mapping for a character Returns the character itself if no mapping exists

    to_titlecase

    source
    fn to_titlecase(c : Char) -> String

    Get the full titlecase mapping for a character Returns a String containing one or more titlecase characters For example, 'ß' (U+00DF) maps to "Ss", and 'ffi' (U+FB03) maps to "Ffi"

    to_uppercase

    source
    fn to_uppercase(c : Char) -> String

    Get the full uppercase mapping for a character Returns a String containing one or more uppercase characters For example, 'ß' (U+00DF) maps to "SS", and 'ffi' (U+FB03) maps to "FFI"

    Source Files