Unicode grapheme cluster segmentation library for MoonBit (UAX #29)

unicode

grapheme

segmentation

text

uax29

cluster

emoji

moon add kawaz/grapheme@0.10.4

Download zip

Author

kawaz

Version

0.10.4

License

MIT

Repository

github.com/kawaz/grapheme.mbt

Last updated

last month

Downloads

18K

README

English | 日本語

#grapheme

"👨‍👩‍👧‍👦".length() returns 11 in MoonBit — this library makes it return 1.

#Overview

MoonBit's String uses UTF-16 internal representation, so length() and str[i] operate at the UTF-16 code unit level. This library provides APIs for safely manipulating strings at the grapheme cluster level (the unit humans perceive as a single "character"), based on the default extended grapheme cluster rules in UAX #29 (Unicode Text Segmentation). Locale-specific tailored rules are not supported.

Zero dependencies
Backends: wasm-gc, js, native
Bundle size: wasm-gc ~25 KB / js ~59 KB / native ~56 KB

Layer	Problem	Solution
L1: UTF-16 encoding	str[i] operates at code unit level	MoonBit core iter()
L2: Grapheme cluster	Composite emoji span multiple code points	This library
L3: Display width	Full-width / half-width display widths	rami3l/unicodewidth

All GB rules (GB3-GB13, GB999) from Unicode 17.0.0 are implemented as a state machine, passing all 766 official test cases.

#Install

moon add kawaz/grapheme

Add the dependency to your package's moon.pkg.json:

{
  "import": ["kawaz/grapheme"]
}

#Usage

// Correct grapheme cluster counting
let family = @grapheme.graphemes("👨‍👩‍👧‍👦")
println(family.length())  // 1

// Split, access, and slice
let view = @grapheme.graphemes("Hello🇯🇵World")
println(view.length())  // 11 (H,e,l,l,o,🇯🇵,W,o,r,l,d)
println(view[5].to_string())  // "🇯🇵"
println(view[1:3].to_string())  // "el" (slice)

// Iteration
for cluster in view {
  println(cluster)
}

// Lazy evaluation: fast when you only need the first few clusters
let first = @grapheme.grapheme_iter("very long text...").head()

graphemes() pre-scans the entire string to provide random access and slicing. O(n) full scan + O(k) memory (k = cluster count).

grapheme_iter() starts in O(1) with no pre-scan. Use it when you only need the first N clusters.

#API

API Documentation

Note: == comparison is based on code point sequences. Unicode normalization (NFC/NFD) is not considered, so precomposed and decomposed forms of the same character are treated as different GraphemeViews.

#Features

UAX #29 Grapheme Cluster Break state machine implementation
Extended_Pictographic property support
Composite emoji support (ZWJ sequences, flags, skin tone modifiers)
Published on mooncakes.io
Two-level lookup table for O(1) property determination
Safe access (get), Show/Eq/Hash traits, is_empty, to_string
Slice operations (view[1:3])
Extended iteration (rev_iter, iter2, grapheme_indices)
Lazy iterator grapheme_iter() — up to 64x faster for early-break use cases

#Unicode Version

Target: Unicode 17.0.0

#License

MIT License - Yoshiaki Kawazu (@kawaz)

#
GraphemeView

source

pub struct GraphemeView {
  // private fields
}

A view type that holds Unicode grapheme cluster boundary information. Each grapheme cluster can be accessed as a zero-copy slice of the original String. Supports slicing via cluster_start and cluster_end indices into the boundaries array.

impl Eq for GraphemeView

fn equal(self : GraphemeView, other : GraphemeView) -> Bool

source

Two GraphemeViews are equal if they have the same number of clusters and each corresponding cluster has the same string content. Compares by code point sequence, not Unicode canonical equivalence (NFC/NFD).

impl Hash for GraphemeView

impl Show for GraphemeView

#
GraphemeView::get

source

fn GraphemeView::get(self : GraphemeView, i : Int) -> StringView?

Returns the i-th grapheme cluster as an Option, or None if out of range.

#
GraphemeView::grapheme_indices

source

fn GraphemeView::grapheme_indices(self : GraphemeView) -> Iter[(Int, Int, StringView)]

Iterates over grapheme clusters with their UTF-16 code unit offsets. Yields (start_offset, end_offset, cluster) for each grapheme cluster. Offsets are compatible with String indexing methods. Note: For sliced GraphemeViews, offsets refer to positions in the original source string.

#
GraphemeView::is_empty

source

fn GraphemeView::is_empty(self : GraphemeView) -> Bool

Returns true if there are no grapheme clusters.

#
GraphemeView::iter

source

fn GraphemeView::iter(self : GraphemeView) -> Iter[StringView]

Iterates over grapheme clusters in order.

#
GraphemeView::iter2

source

fn GraphemeView::iter2(self : GraphemeView) -> Iter2[Int, StringView]

Iterates over grapheme clusters with their 0-based cluster indices (not byte offsets).

#
GraphemeView::length

source

fn GraphemeView::length(self : GraphemeView) -> Int

Returns the number of grapheme clusters.

#
GraphemeView::op_as_view

source

fn GraphemeView::op_as_view(self : GraphemeView, start? : Int, end? : Int) -> GraphemeView

Returns a sliced GraphemeView. Enables view[start:end] syntax. Negative start is clamped to 0. End beyond length is clamped to length. If end < start, an empty view is returned.

#
GraphemeView::op_get

source

fn GraphemeView::op_get(self : GraphemeView, i : Int) -> StringView

Returns the i-th grapheme cluster as a StringView. Panics if i is not in the range 0 <= i < length().

#
GraphemeView::rev_iter

source

fn GraphemeView::rev_iter(self : GraphemeView) -> Iter[StringView]

Iterates over grapheme clusters in reverse order.

#
GraphemeView::to_string

source

fn GraphemeView::to_string(self : GraphemeView) -> String

Returns the string content of this view (may be a substring if sliced).

#
grapheme_iter

source

fn grapheme_iter(s : String) -> Iter[StringView]

Returns a lazy iterator that yields grapheme clusters one at a time without preprocessing the entire string. Use this when you only need the first few clusters of a long string. For random access or repeated iteration, use graphemes() instead.

#
graphemes

source

fn graphemes(s : String) -> GraphemeView

Returns a GraphemeView that splits the string into grapheme cluster units. Implements UAX #29 GB rules as a state machine. Note: This function scans the entire string upfront (O(n) preprocessing). For lazy evaluation, use grapheme_iter() instead.