🥮 mooncakes.io

    unicode-segmentation

    Unicode Text Segmentation for MoonBit

    unicode
    segmentation
    uax29
    grapheme
    uax14
    linebreak
    Download zip
    Version
    0.7.1
    License
    Apache-2.0
    Repository
    Last updated
    8 days ago
    Downloads
    97

    #unicode-segmentation

    Unicode 18.0.0 text segmentation for MoonBit: extended grapheme clusters, word and sentence boundaries (UAX #29), and default line break opportunities (UAX #14, revision 57).

    moon add moonbit-community/unicode-segmentation

    Import "moonbit-community/unicode-segmentation" @segmentation in moon.pkg.

    #Words and sentences

    for segment in @segmentation.word_segments("Hello, world! 42") {
      // "Hello", ",", " ", "world", "!", " ", "42"
    }
    for word in @segmentation.words("Hello, world! 42") {
      // "Hello", "world", "42"
    }
    for sentence in @segmentation.sentences("Hello! Next.") {
      // "Hello! ", "Next."
    }

    Entry pointCursorContents
    word_segments(StringView)WordSegmentsAll default UAX #29 word segments, including whitespace and punctuation.
    words(StringView)WordsWord segments containing at least one Unicode Alphabetic character or General_Category=Number character (Nd, Nl, or No).
    sentences(StringView)SentencesAll default UAX #29 sentence segments, retaining closing punctuation, trailing whitespace, and paragraph separators as specified by the rules.

    Each opaque cursor provides next() -> StringView and iter() -> Iter[StringView]. Both consume the same cursor and can be interleaved. next() returns an empty view at the input's end on every call after exhaustion; empty input produces no segments. Returned views retain UTF-16 offsets in the backing string and do not copy the text.

    word_segments and sentences cover the input exactly. words filters whole segments, so it retains internal punctuation in words such as can't and numbers such as 3.14, while omitting punctuation, whitespace, and emoji segments without Alphabetic or Number characters. A digit keycap such as 1️⃣ is retained because it contains a digit. This filter is a convenience policy, not a linguistic test for whether a segment is a dictionary word.

    The input view is segmented independently of surrounding text. Valid surrogate pairs stay together; unpaired surrogates use the default Other property. Word and sentence rules are evaluated independently of grapheme segmentation; their boundaries are not forced to cluster ends.

    Construction takes O(n) time and space to prepare boundary context and store segment ends. Complete traversal takes O(n) time; words additionally scans each segment to apply its filter, still in O(n) total time. No locale tailoring, abbreviation dictionary, or dictionary-based word segmentation is applied. In particular, default word boundaries do not provide linguistic Chinese or Thai word recognition.

    All 1,944 WordBreakTest.txt and 512 SentenceBreakTest.txt cases from Unicode 18.0.0 run directly from tests/data/ in tests/segmentation_tests.

    #Graphemes with line break information

    let graphemes = @segmentation.graphemes("ab c")
    for grapheme in graphemes {
      let opportunity = graphemes.line_break()
      // "a" -> Prohibited, "b" -> Prohibited,
      // " " -> Allowed, "c" -> Mandatory
    }

    graphemes(StringView) -> Graphemes creates a forward cursor over extended grapheme clusters. next() -> StringView returns an empty view at exhaustion; iter() -> Iter[StringView] consumes the same cursor.

    Graphemes::line_break() -> BreakOpportunity queries the boundary after the last returned cluster. The first query after a cluster prepares line breaking context for the whole input in O(n) time and space. Subsequent queries scan forward in O(n) total time over the traversal. Querying repeatedly does not advance the grapheme cursor, and queries may be skipped between clusters. Traversal without this query does not run the line breaking algorithm.

    The query exposes only opportunities at complete cluster ends. Default UAX #14 can allow a break inside a cluster, for example between SPACE and a combining mark. Such opportunities are not exposed or moved to the end of the cluster. Grapheme segmentation and the default line breaking rules remain independent; this is not the per-cluster class tailoring from UAX #14 §8.

    #Independent line break cursor

    let breaks = @segmentation.line_breaks("Hello world!")
    for segment in breaks {
      let opportunity = breaks.opportunity()
      // "Hello " -> Allowed, "world!" -> Mandatory
    }

    line_breaks(StringView) -> LineBreaks exposes all default UAX #14 opportunities. next() -> StringView and iter() -> Iter[StringView] consume the same cursor. opportunity() -> BreakOpportunity reports the boundary at the end of the most recently returned nonempty segment. Query the kind while processing that segment; collecting the views does not retain their kinds.

    • Segments are contiguous views covering the input exactly, retaining spaces and newline characters. They may end inside a grapheme cluster.
    • Views retain UTF-16 offsets in their backing string. The supplied input view is processed independently of surrounding text.
    • Empty input produces no segments. Exhausted cursors keep returning an empty view at the end of the input.
    • Construction takes O(n) time and space to prepare character context. Iteration scans that context once without copying the text.
    • Valid surrogate pairs stay together. Unpaired surrogates use SG -> AL.

    #Break kinds and cursor state

    BreakOpportunity has Prohibited, Allowed, and Mandatory variants. Before the first result (including empty input), both cursors' break queries return Prohibited. After a nonempty result they describe its trailing boundary. Exhaustion preserves the last kind. Nonempty input ends with Mandatory, even without a newline; this does not insert an extra empty line. Every nonempty LineBreaks segment ends with Allowed or Mandatory.

    The algorithm applies no locale tailoring or dictionary segmentation. It finds legal opportunities; selecting actual lines based on font measurements and available width is the caller's job. The separate unicode-data.line_break(Char) query still returns the original UCD property; LB1 resolution happens only within the algorithm.

    All 19,346 cases in the official Unicode 18.0.0 LineBreakTest.txt run in the workspace-only tests/linebreak_tests package. The same cases also check the combined grapheme cursor against the opportunities at its cluster ends. The existing GraphemeBreakTest.txt suite checks UAX #29 independently.

    Stateless boundary queries are deferred. The repository's boundary API design note records the proposed signatures, context requirements, and representation experiments for future work.

    moon test --target native segmentation tests/linebreak_tests tests/segmentation_tests

    Specifications: UAX #14, UAX #29.

    BreakOpportunity

    source
    pub(all) enum BreakOpportunity {
      Prohibited
      Allowed
      Mandatory
    } derive(Eq, 
    Debug    )

    Whether a line may end at a text boundary. Mandatory includes the end of nonempty text, even when the text does not end in a newline character.

    BreakOpportunity::equal

    source

    Graphemes

    source
    type Graphemes

    A single-pass cursor over the extended grapheme clusters of a text, the user-perceived characters defined by Unicode Text Segmentation (UAX #29).

    Create one with graphemes. Use it in a for .. in loop, or call next directly to avoid allocating an Option per cluster.

    Graphemes::iter

    source
    fn Graphemes::iter(self : Graphemes) -> Iter[StringView]

    Iterate over the remaining clusters. The iterator advances this cursor, so clusters it yields are not returned by next again.

    Graphemes::line_break

    source
    fn Graphemes::line_break(self : Graphemes) -> BreakOpportunity

    Return the default UAX #14 break kind after the last grapheme returned by next() or iter(). Before the first grapheme, including for empty input, returns Prohibited. Exhaustion preserves the last kind (Mandatory for nonempty text). Repeated queries do not advance the grapheme cursor.

    Only complete grapheme ends are queried; default line break opportunities inside a cluster are not exposed. This does not change UAX #29 segmentation or recompute line breaking classes per cluster.

    The first query after a grapheme prepares line breaking context for the whole input in O(n) time and space. Subsequent queries scan that context forward, taking O(n) total time over a complete traversal. Traversal without this query does not initialize or run the line breaking algorithm.

    test {
      let graphemes = @segmentation.graphemes("ab c")
      let kinds = []
      for grapheme in graphemes {
        kinds.push((grapheme.to_owned(), graphemes.line_break()))
      }
      debug_inspect(
        kinds,
        content=(
          #|[
          #|  ("a", Prohibited),
          #|  ("b", Prohibited),
          #|  (" ", Allowed),
          #|  ("c", Mandatory),
          #|]
        ),
      )
    }

    Graphemes::next

    source
    fn Graphemes::next(self : Graphemes) -> StringView

    Return the next cluster and advance past it, or return an empty view once the text is exhausted. A cluster is never empty, so an empty result always means the end.

    test {
      let clusters = @segmentation.graphemes("e\u{301}x")
      inspect(
        clusters.next(),
        content=(
          #|é
        ),
      )
      inspect(
        clusters.next(),
        content=(
          #|x
        ),
      )
      inspect(
        clusters.next().is_empty(),
        content=(
          #|true
        ),
      )
    }

    LineBreaks

    source
    type LineBreaks

    A forward cursor over text segments ending at default UAX #14 opportunities.

    LineBreaks::iter

    source
    fn LineBreaks::iter(self : LineBreaks) -> Iter[StringView]

    Iterate over the remaining segments, advancing this same cursor. Query opportunity() in the loop body for the segment just yielded. Collected views retain their text and positions, but do not store the break kind.

    LineBreaks::next

    source
    fn LineBreaks::next(self : LineBreaks) -> StringView

    Return the next segment, or an empty view at the end of the input once exhausted. The returned view retains its offsets in the backing string. Empty results do not change opportunity().

    LineBreaks::opportunity

    source
    fn LineBreaks::opportunity(self : LineBreaks) -> BreakOpportunity

    The break kind at the end of the last nonempty segment returned by next() or iter(). Before the first segment, including for empty input, returns Prohibited. Exhaustion preserves the last kind; querying does not advance.

    Sentences

    source
    type Sentences

    A forward cursor over default UAX #29 sentence segments. Create with sentences; trailing spaces and punctuation remain in the returned views.

    Sentences::iter

    source
    fn Sentences::iter(self : Sentences) -> Iter[StringView]

    Iterate over remaining segments, consuming the same cursor as next.

    Sentences::next

    source
    fn Sentences::next(self : Sentences) -> StringView

    Return the next segment, or an empty view at the input's end on exhaustion.

    WordSegments

    source
    type WordSegments

    A forward cursor over all default UAX #29 word segments, including spaces and punctuation. Create with word_segments.

    WordSegments::iter

    source
    fn WordSegments::iter(self : WordSegments) -> Iter[StringView]

    Iterate over the remaining segments, consuming the same cursor as next.

    WordSegments::next

    source
    fn WordSegments::next(self : WordSegments) -> StringView

    Return the next segment, or an empty view at the input's end on exhaustion.

    Words

    source
    type Words

    A forward cursor over word segments containing a Unicode Alphabetic or Number character. Create with words.

    Words::iter

    source
    fn Words::iter(self : Words) -> Iter[StringView]

    Iterate over the remaining words, consuming the same cursor as next.

    Words::next

    source
    fn Words::next(self : Words) -> StringView

    Return the next word, skipping other segments. Returns an empty view at the input's end once exhausted, including when the input has no words.

    graphemes

    source
    fn graphemes(text : StringView) -> Graphemes

    Split text into extended grapheme clusters.

    Each cluster is a view into text, so StringView::start_offset gives its position in the underlying string. The clusters are contiguous and cover text exactly. text is segmented on its own: its start is a boundary even if the underlying string continues before it. Unpaired surrogates have Grapheme_Cluster_Break=Other; valid pairs stay together.

    test {
      let clusters = []
      for cluster in @segmentation.graphemes("e\u{301}🇺🇳x") {
        clusters.push(cluster.to_owned())
      }
      debug_inspect(
        clusters,
        content=(
          #|["é", "🇺🇳", "x"]
        ),
      )
    }

    line_breaks

    source
    fn line_breaks(text : StringView) -> LineBreaks

    Find the default Unicode 18.0.0 UAX #14 line break opportunities.

    Segments are contiguous views covering the input, retaining spaces and newline characters. The view is processed independently of its backing string. Default line breaks may fall inside extended grapheme clusters; use Graphemes::line_break to query only the ends of complete clusters.

    Nonempty input ends with one Mandatory boundary; empty input yields nothing. Unpaired surrogates are resolved as SG -> AL; valid pairs stay together. Construction prepares character context in O(n) time and space; iteration scans it once without copying the text. No locale tailoring, dictionary segmentation, or width-based wrapping is applied. The unicode-data line_break property query remains unresolved.

    test {
      let breaks = @segmentation.line_breaks("Hello world!")
      inspect(breaks.next(), content="Hello ")
      debug_inspect(breaks.opportunity(), content="Allowed")
      inspect(breaks.next(), content="world!")
      debug_inspect(breaks.opportunity(), content="Mandatory")
      assert_true(breaks.next().is_empty())
    }

    sentences

    source
    fn sentences(text : StringView) -> Sentences

    Split text at the default Unicode 18.0.0 sentence boundaries, retaining every character. Closing punctuation, trailing spaces, and paragraph separators are included according to UAX #29. No abbreviation dictionary or locale tailoring is applied. The supplied view is an independent text. Construction takes O(n) time and space; traversal does not copy text. Unpaired surrogates have Sentence_Break=Other; valid pairs stay together.

    test {
      debug_inspect(
        @segmentation.sentences("Hello! Next.")
        .iter()
        .map(s => s.to_owned())
        .collect(),
        content="[\"Hello! \", \"Next.\"]",
      )
    }

    word_segments

    source
    fn word_segments(text : StringView) -> WordSegments

    Split text at the default Unicode 18.0.0 word boundaries. Segments are contiguous views covering the input exactly. The input view is processed independently of surrounding text, without dictionary or locale tailoring. Construction takes O(n) time and space; traversal does not copy the text. Unpaired surrogates have Word_Break=Other; valid pairs stay together.

    test {
      debug_inspect(
        @segmentation.word_segments("Hello, world!")
        .iter()
        .map(s => s.to_owned())
        .collect(),
        content="[\"Hello\", \",\", \" \", \"world\", \"!\"]",
      )
    }

    words

    source
    fn words(text : StringView) -> Words

    Filter default word segments to those containing at least one Alphabetic character or a character in General_Category=Number (Nd, Nl, or No). The entire matching segment is retained, including internal punctuation. This filtering policy is a convenience over UAX #29 word boundaries, not dictionary word recognition. Construction takes O(n) time and space; traversal scans the segments without copying text.

    test {
      debug_inspect(
        @segmentation.words("Hello, world! 42")
        .iter()
        .map(s => s.to_owned())
        .collect(),
        content="[\"Hello\", \"world\", \"42\"]",
      )
    }

    Source Files