🥮 mooncakes.io

jieba

Chinese word segmentation based on jieba

jieba
segmentation
search
chinese
moon add colmugx/jieba@0.1.1
Download zip
Author
Version
0.1.1
License
Apache-2.0
Last updated
2 months ago
Downloads
46
README

#colmugx/jieba

A MoonBit Chinese word segmentation package built on jieba data and behavior.

#Features

  • Supports wasm, wasm-gc, js, native
  • Accurate mode, full mode, search engine mode
  • DAG + dynamic programming for maximum-probability paths
  • BMES HMM + Viterbi for unknown word recognition
  • Default dictionary, user dictionary, dynamic add/remove words, suggested word frequencies
  • Unicode character-position tokenize
  • No Python dependency at runtime, no filesystem dependency

#Usage

fn main raise {
  let tokenizer = @jieba.Tokenizer()
  let words = tokenizer.cut(
    "小明硕士毕业于中国科学院计算所",
    mode=Search,
  )
  println(words.join(" / "))
}

Custom dictionaries work directly with text — no binding to platform file APIs:

fn main raise {
  let tokenizer = @jieba.Tokenizer()
  tokenizer.load_user_dictionary(
    (
      #|生成式人工智能 100000 nz
      #|向量数据库 100000 nz
      #|混合检索 100000 nz
    ),
  )
  println(tokenizer.cut("生成式人工智能与向量数据库的混合检索"))
}

#Architecture

The project maintains three layers:

  1. internal/core: trie, DAG, dynamic programming, HMM, Viterbi — accepts only MoonBit data structures, no platform capabilities.
  2. internal/adaptor: supplies versioned embedded jieba dictionary and HMM data to core. Future external storage, network, or platform file API integrations should also be adapted at this layer.
  3. Root package jieba: stable public API and error conversion.

Instead of the Python approach of putting all prefixes into a hash table, the dictionary uses a character trie to avoid storing duplicate strings for prefixes.

#Examples

examples/search covers several search scenarios:

  • E-commerce product title recall
  • Technical documentation with custom domain terms
  • Address and organization name search
  • Building inverted index positions using token offsets

Run:

moon -C examples run search --target native
moon -C examples run search --target js
moon -C examples run search --target wasm-gc

#Data Source

The default dictionary and HMM parameters are pinned from fxsjy/jieba v0.42.1. This project uses and re-encodes the default dictionary as well as the HMM initial, transition, and emission probabilities from that release. These data are embedded as MoonBit source in internal/adaptor; no Python, jieba package, or external data files are needed for distribution or runtime.

#
JiebaError

source
pub(all) suberror JiebaError {
  InvalidDictionary(String)
} derive(Eq, 
Debug)

#
CutMode

source
pub(all) enum CutMode {
  Accurate
  Full
  Search
} derive(Eq, 
Debug)

#
Token

source
pub(all) struct Token {
  word : String
  start : Int
  end : Int
} derive(Eq, 
Debug)

#
Tokenizer

source
pub struct Tokenizer {
  dictionary : 
Dictionary
  hmm_model : 
HmmModel?
}
fn Tokenizer::Tokenizer() -> Tokenizer raise JiebaError

#
Tokenizer::add_word

source
fn Tokenizer::add_word(self : Tokenizer, word : String, frequency? : Int, tag? : String) -> Int

#
Tokenizer::contains

source
fn Tokenizer::contains(self : Tokenizer, word : String) -> Bool

#
Tokenizer::cut

source
fn Tokenizer::cut(self : Tokenizer, sentence : String, mode? : CutMode, hmm? : Bool) -> Array[String]

#
Tokenizer::delete_word

source
fn Tokenizer::delete_word(self : Tokenizer, word : String) -> Unit

#
Tokenizer::from_dictionary

source
fn Tokenizer::from_dictionary(dictionary : String) -> Tokenizer raise JiebaError

#
Tokenizer::from_resources

source
fn Tokenizer::from_resources(dictionary : String, hmm_model? : String) -> Tokenizer raise JiebaError

Creates a tokenizer from backend-independent resource text.

Platform-specific file, network, or database access should happen outside this package and pass the resulting text through this constructor.

#
Tokenizer::load_user_dictionary

source
fn Tokenizer::load_user_dictionary(self : Tokenizer, dictionary : String) -> Unit raise JiebaError

#
Tokenizer::suggest_frequency

source
fn Tokenizer::suggest_frequency(self : Tokenizer, word : String) -> Int

#
Tokenizer::suggest_split_frequency

source
fn Tokenizer::suggest_split_frequency(self : Tokenizer, segments : Array[String]) -> Int

#
Tokenizer::tokenize

source
fn Tokenizer::tokenize(self : Tokenizer, sentence : String, mode? : CutMode, hmm? : Bool) -> Array[Token]

#
JIEBA_VERSION

source
let JIEBA_VERSION : String

Source Files

Powered by MoonBit

Site sourceReport issuePackagesBuild queueSkillsStatistics

© 2026 mooncakes.io