🥮 mooncakes.io

    moonseek

    Embedded full-text search engine: inverted index, BM25 ranking, Chinese/English tokenization, crash-safe segments, runs on native/WASM/JS.

    search
    full-text
    index
    bm25
    chinese
    wasm
    Download zip
    Version
    0.1.0
    License
    Apache-2.0
    Repository
    Last updated
    1 hour ago
    Downloads
    2

    #moonseek

    Embedded full-text search engine in pure MoonBit. Inverted index, BM25 ranking, dictionary-free Chinese tokenization, crash-safe snapshots — runs on native, WebAssembly and JavaScript.

    纯 MoonBit 实现的嵌入式全文搜索引擎：倒排索引、BM25 排序、免词典中文分词、崩溃安全快照，支持 native / WASM / JS 三端。

    CI License

    #Why moonseek

    MoonBit's ecosystem has HTTP servers, crypto and database drivers — but no tantivy/Lucene-class embedded search engine. moonseek fills that gap with a single dependency-light library that works everywhere MoonBit compiles, including inside the browser via WasmGC.

    #Features

    • Inverted index with per-field weights (e.g. title ×3, body ×1)
    • BM25 ranking with configurable k1 / b
    • Chinese + English tokenization: Latin words with stop-word removal; CJK overlapping bigrams (Lucene CJKAnalyzer strategy) — no dictionary needed
    • Rich query syntax: term, "exact phrase", AND / OR / NOT, field:term, prefix*, parentheses, implicit AND
    • Crash-safe persistence: versioned snapshot + CRC-32 checksum; atomic save (temp file → fsync → rename); torn/corrupt files are rejected, never silently loaded
    • Portable: pure MoonBit core (no FFI in the engine); native file store via moonbitlang/async; browser export via WasmGC + js-string builtins
    • Document lifecycle: add, replace (same id), delete

    #Quick start

    let index = @moonseek.Index::new([
      { name: "title", weight: 3.0 },
      { name: "body", weight: 1.0 },
    ])

    index.add({
      id: "doc1",
      fields: [
        ("title", "全文搜索引擎"),
        ("body", "倒排索引是搜索引擎的核心，BM25 负责排序。"),
      ],
    })
    index.add({
      id: "doc2",
      fields: [("title", "MoonBit"), ("body", "A fast language for WebAssembly")],
    })

    let hits = index.search("搜索引擎")        // [{ id: "doc1", score: … }]
    let hits2 = index.search("title:moonbit")  // field-scoped
    let hits3 = index.search("\"memory safety\" OR wasm*")  // phrase + prefix

    // Persist (native)
    index.save("index.msk")
    let restored = @moonseek.Index::load("index.msk")

    // Or portable bytes (works on every backend)
    let image : Bytes = index.to_bytes()
    let restored2 = @moonseek.Index::from_bytes(image)

    #CLI

    moon run cmd/main -- index ./docs --index index.msk   # index .txt/.md files
moon run cmd/main -- search "搜索引擎" --index index.msk
moon run cmd/main -- search "title:rust" --limit 5
moon run cmd/main -- info --index index.msk

    #Browser demo

    A complete search engine running inside your browser — no server, no network:

    moon build --target wasm-gc
cp _build/wasm-gc/debug/build/web/web.wasm demo/web.wasm
cd demo && python -m http.server 8000
# open http://localhost:8000 (Chrome / Edge with WasmGC + js-string builtins)

    The demo indexes 15 documents (MoonBit docs + 论语/道德经) in ~6 ms and answers Chinese, English, phrase and field queries with highlighting.

    #Performance

    moon bench --target native --release, 1000 synthetic documents (~80 tokens each), AMD Ryzen 7 7435H:

    OperationTime
    Index 1000 docs~89 ms
    Term query (top-10)~1.0 ms
    Phrase query (top-10)~2.9 ms
    Snapshot save+load~30 ms

    #Architecture

    token.mbt     Analyzer: Latin words + CJK bigrams, positions
index.mbt     Inverted index, field stats, BM25 scoring
query.mbt     Lexer → recursive-descent parser → evaluator
codec.mbt     Varint / UTF-8 / double / CRC-32 primitives
snapshot.mbt  MSK1 format: versioned payload + checksum
store.mbt     Native atomic file persistence (async/fs)
web/          Browser exports (#export_name, js-string builtins)
cmd/main/     CLI: index / search / info
demo/         Single-page browser demo

    See docs/architecture.md for design details and the snapshot format specification.

    #Testing

    moon test                    # 43 tests (native), 39 (wasm-gc / js)
moon test --target wasm-gc
moon test --target js
moon bench --release         # benchmarks

    Tests cover tokenization, boolean/phrase/prefix queries, BM25 ordering, snapshot roundtrips, corruption/truncation detection, and crash-safe file persistence.

    #Roadmap

    #License

    Apache-2.0. Built for the MoonBit open-source ecosystem.

    CodecError

    source
    pub(all) suberror CodecError {
      UnexpectedEof
      VarintOverflow
      InvalidUtf8
      BadMagic
      BadVersion(Int)
      CrcMismatch(UInt, UInt)
    } derive(Eq, 
    Debug    )

    Decoding errors (truncated or malformed data).

    QueryError

    source
    pub(all) suberror QueryError {
      EmptyQuery
      UnterminatedPhrase
      UnexpectedToken(String)
      UnknownField(String)
    } derive(Eq, Show, 
    Debug    )

    Query syntax errors.

    StoreError

    source
    pub(all) suberror StoreError {
      IoError(String)
      CorruptIndex(String)
    } derive(Eq, Show, 
    Debug    )

    Errors from the store layer.

    Analyzer

    source
    pub(all) struct Analyzer {
      lowercase : Bool
      remove_stop_words : Bool
      cjk_bigram : Bool
    } derive(Eq, 
    Debug    )

    Text analyzer configuration.

    Analyzer::analyze

    source
    fn Analyzer::analyze(self : Analyzer, text : String) -> Array[Token]

    Analyze text into a stream of tokens. Latin words are split on non-word characters; CJK runs become overlapping bigrams (single characters are emitted for runs of length 1).

    Analyzer::default

    source
    fn Analyzer::default() -> Analyzer

    Bm25Params

    source
    pub(all) struct Bm25Params {
      k1 : Double
      b : Double
    } derive(Eq, 
    Debug    )

    BM25 parameters (Robertson/Sparck Jones defaults).

    Bm25Params::default

    source
    fn Bm25Params::default() -> Bm25Params

    Document

    source
    pub(all) struct Document {
      id : String
      fields : Array[(String, String)]
    } derive(Eq, 
    Debug    )

    A document to be indexed. id is caller-assigned and must be unique within the index; re-adding an existing id replaces the old document.

    FieldSchema

    source
    pub(all) struct FieldSchema {
      name : String
      weight : Double
    } derive(Eq, 
    Debug    )

    Field schema: name + scoring weight (e.g. title=3.0, body=1.0).

    FieldStats

    source
    type FieldStats

    Per-field statistics needed by BM25.

    Hit

    source
    pub(all) struct Hit {
      id : String
      score : Double
    } derive(Eq, 
    Debug    )

    One search hit.

    Index

    source

    An in-memory inverted index over a fixed field schema.

    Index::add

    source
    fn Index::add(self : Index, doc : Document) -> Unit

    Add or replace a document. Unknown field names abort.

    Index::delete

    source
    fn Index::delete(self : Index, id : String) -> Bool

    Delete a document by external id. Returns true if it existed.

    Index::doc_count

    source
    fn Index::doc_count(self : Index) -> Int

    Number of live documents.

    Index::doc_id

    source
    fn Index::doc_id(self : Index, num : Int) -> String?

    Look up the external id of an internal doc number.

    Index::from_bytes

    source
    fn Index::from_bytes(data : Bytes) -> Index raise CodecError

    Restore an index from a byte image. Raises on bad magic, unknown version, truncation, or checksum mismatch.

    Index::load

    source
    async fn Index::load(path : String) -> Index raise StoreError

    Load an index from path. Raises if the file is missing or corrupt.

    Index::load_or_create

    source
    async fn Index::load_or_create(path : String, schema : Array[FieldSchema]) -> Index raise StoreError

    Load an index from path, or create a fresh one if the file does not exist. A corrupt file is still an error (never silently discarded).

    Index::new

    source
    fn Index::new(schema : Array[FieldSchema], analyzer? : Analyzer, bm25? : Bm25Params) -> Index

    Create an empty index. Panics on duplicate field names.

    Index::postings_of

    source
    fn Index::postings_of(self : Index, field : String, term : String) -> Array[Posting]

    Postings of a term in a field (empty if absent).

    Index::save

    source
    async fn Index::save(self : Index, path : String) -> Unit raise StoreError

    Save the index to path atomically (write temp + fsync + rename).

    Index::search

    source
    fn Index::search(self : Index, query : String, limit? : Int) -> Array[Hit] raise QueryError

    Search the index. limit caps the number of hits (0 = unlimited). Results are sorted by descending score, ties broken by document id.

    Index::term_count

    source
    fn Index::term_count(self : Index) -> Int

    Number of distinct terms across all fields.

    Index::terms_of

    source
    fn Index::terms_of(self : Index, field : String) -> Array[String]

    All terms in a field, sorted lexicographically (for prefix queries).

    Index::to_bytes

    source
    fn Index::to_bytes(self : Index) -> Bytes

    Serialize the index to a portable byte image.

    Parser

    source
    type Parser

    Recursive-descent parser. Grammar: or_expr := and_expr (OR and_expr)* and_expr := not_expr ((AND)? not_expr)* (implicit AND) not_expr := primary (NOT primary)* primary := '(' or_expr ')' | phrase | term

    Posting

    source
    pub(all) struct Posting {
      doc : Int
      freq : Int
      positions : Array[Int]
    } derive(Eq, 
    Debug    )

    One posting: which document, how many times, and where.

    QToken

    source

    Tokenize a query string into raw pieces: words, quoted phrases, operators (AND OR NOT), parentheses, and field prefixes.

    Query

    source
    pub(all) enum Query {
      Term(String, String?)
      Phrase(Array[String], String?)
      Prefix(String, String?)
      And(Query, Query)
      Or(Query, Query)
      Not(Query, Query)
      MatchAll
    } derive(Eq, 
    Debug    )

    A parsed query node.

    Reader

    source
    pub struct Reader {
      data : Bytes
      pos : Int
    }

    Cursor over a byte slice for decoding.

    Reader::new

    source
    fn Reader::new(data : Bytes) -> Reader

    Reader::read_byte

    source
    fn Reader::read_byte(self : Reader) -> Byte raise CodecError

    Reader::read_double

    source
    fn Reader::read_double(self : Reader) -> Double raise CodecError

    Reader::read_string

    source
    fn Reader::read_string(self : Reader) -> String raise CodecError

    Reader::read_u32_le

    source
    fn Reader::read_u32_le(self : Reader) -> UInt raise CodecError

    Reader::read_varint

    source
    fn Reader::read_varint(self : Reader) -> Int raise CodecError

    Reader::remaining

    source
    fn Reader::remaining(self : Reader) -> Int

    Token

    source
    pub(all) struct Token {
      term : String
      position : Int
    } derive(Eq, Show, 
    Debug    )

    A token emitted by the analyzer: the normalized term plus its position in the token stream (used for phrase queries).

    crc32

    source
    fn crc32(data : Bytes) -> UInt

    CRC-32 (IEEE) of a byte slice.

    parse_query

    source
    fn parse_query(input : String, analyzer? : Analyzer, fields? : Array[String]) -> Query raise QueryError

    Parse a query string against the index's schema.

    tokenize

    source
    fn tokenize(text : String) -> Array[Token]

    Convenience: analyze with the default analyzer.

    write_double

    source
    fn write_double(buf : Array[Byte], value : Double) -> Unit

    write_string

    source
    fn write_string(buf : Array[Byte], s : String) -> Unit

    write_u32_le

    source
    fn write_u32_le(buf : Array[Byte], value : UInt) -> Unit

    Append a little-endian uint32 (used for the trailing checksum).

    write_varint

    source
    fn write_varint(buf : Array[Byte], value : Int) -> Unit

    Append an unsigned varint (LEB128) to the buffer. Every byte except the last has the continuation bit (0x80) set.

    Source Files

    Powered by MoonBit

    Site sourceReport issueModulesSkillsBuild queueStatistics

    © 2026 mooncakes.io