unicode-segmentation

    Unicode Text Segmentation for MoonBit

    unicode
    segmentation
    uax29
    grapheme
    Download zip
    Version
    0.6.0
    License
    Apache-2.0
    Last updated
    last month
    Downloads
    101

    No dependencies

    #unicode-segmentation

    Unicode Text Segmentation for MoonBit, targeting UAX #29 and Unicode 18.0.0. It currently provides extended grapheme cluster boundaries.

    moon add moonbit-community/unicode-segmentation

    See the repository root README for API documentation and examples.

    Graphemes

    type Graphemes

    A single-pass cursor over the extended grapheme clusters of a text, the user-perceived characters defined by Unicode Text Segmentation (UAX #29).

    Create one with graphemes. Use it in a for .. in loop, or call next directly to avoid allocating an Option per cluster.

    Graphemes::iter

    fn Graphemes::iter(self : Graphemes) -> Iter[StringView]

    Iterate over the remaining clusters. The iterator advances this cursor, so clusters it yields are not returned by next again.

    Graphemes::next

    fn Graphemes::next(self : Graphemes) -> StringView

    Return the next cluster and advance past it, or return an empty view once the text is exhausted. A cluster is never empty, so an empty result always means the end.

    test {
    let clusters = @segmentation.graphemes("e\u{301}x")
    inspect(
    clusters.next(),
    content=(
    #|é
    ),
    )
    inspect(
    clusters.next(),
    content=(
    #|x
    ),
    )
    inspect(
    clusters.next().is_empty(),
    content=(
    #|true
    ),
    )
    }

    graphemes

    fn graphemes(text : StringView) -> Graphemes

    Split text into extended grapheme clusters.

    Each cluster is a view into text, so StringView::start_offset gives its position in the underlying string. The clusters are contiguous and cover text exactly. text is segmented on its own: its start is a boundary even if the underlying string continues before it.

    test {
    let clusters = []
    for cluster in @segmentation.graphemes("e\u{301}🇺🇳x") {
    clusters.push(cluster.to_owned())
    }
    debug_inspect(
    clusters,
    content=(
    #|["é", "🇺🇳", "x"]
    ),
    )
    }