#moonseek

#Why moonseek

#Features

• Inverted index with per-field weights (e.g. title ×3, body ×1)
• BM25 ranking with configurable k1 / b
• Chinese + English tokenization: Latin words with stop-word removal; CJK overlapping bigrams (Lucene CJKAnalyzer strategy) — no dictionary needed
• Rich query syntax: term, "exact phrase", "a b"~3 (sloppy phrase: up to 3 tokens between terms), AND / OR / NOT, field:term, prefix*, parentheses, implicit AND
• Fuzzy search: Levenshtein / Damerau-Levenshtein edit distance with early-exit bounded DP; BK-tree for sub-linear lookup (13–1780× faster than linear scan on a 24k-term vocabulary)
• Did-you-mean: frequency-weighted spelling correction (suggest)
• Faceted search: exact-match facet fields, AND/OR filters, per-facet value counts, offset pagination (search_faceted)
• Ranking control: configurable sort — score, id, or a numeric field, in either direction, over HTTP as sort=num:year&order=desc and in the library as search_sorted; cursor pagination (search_after); per-term BM25 breakdown (explain). A cross-shard numeric sort is exact: the shards ship the key
• Server-side highlighting: highlight / snippet with HTML escaping, CJK-aware codepoint spans, overlap merging, densest-window extraction
• Stored fields: original texts kept in the index, retrievable via get_document / get_field, survive snapshots
• Segmented index: immutable segments + tiered merge policy + cross-segment search with latest-version-wins (SegmentedIndex)
• Write-ahead log: CRC-framed entries, torn-tail recovery, durable add + atomic checkpoint + crash-safe reopen (native)
• Crash-safe persistence: versioned snapshot + CRC-32 checksum; atomic save (temp file → fsync → rename); torn/corrupt files are rejected, never silently loaded
• Portable: pure MoonBit core (no FFI in the engine); native file store via moonbitlang/async; browser export via WasmGC + js-string builtins. One snapshot format, two backends: an index built by the native binary loads byte-for-byte into the browser's wasm-gc build
• Measured quality, not just latency: NDCG@10 / MRR@10 / Recall@20 over 3,698 real packages with a 6-way ablation (docs/quality.md)
• HTTP service: a REST search API (docs/api.md) with bulk ingest, facets, aggregations, highlighting, explain, suggest and vector queries. The console in console/ drives it
• Durable service: --data-dir journals every accepted write (CRC-framed, data fsync) before applying it and checkpoints atomically, so a restart replays text, facets, numbers, vectors and deletions exactly. One fsync per bulk batch keeps durable ingest within ~1.2–1.3× of in-memory, and concurrent single-document writes share a flush through group commit — 1.9× at 8 writers, 3.4× at 32 over one-fsync-per-request. tools/test-durability.sh and tools/test-durability-concurrent.sh restart the process to prove it
• Distributed cluster: one binary is a node, a shard or a coordinator. Consistent hashing (a fourth shard moves 24% of the corpus, not 75%), write routing with replication to every group member, cross-shard facets, aggregations, sorts and hybrid retrieval that equal a single node, globally consistent scoring (?dfs=true reproduced a single node's ranking exactly, 8/8 queries, zero score difference), POST /_rebalance to grow without a reindex (the plan predicted 762 moves and the executor made 762, ending at exactly the predicted distribution), POST /_repair to bring a member that missed writes back in line, replica failover, and partial answers that name the shards that failed (docs/cluster.md)
• Vector search: pure-MoonBit HNSW with measured recall against brute force (0.996@ef=32, 0.999@ef=64 on 1,000 random 32-d points), plus hybrid BM25 × kNN fusion by reciprocal rank
• Numeric aggregations: terms, stats, histogram and percentiles over a result set, with numeric doc values alongside facets
• Document lifecycle: add, replace (same id), delete

#How it works

flowchart LR
  T[text] --> A[Analyzer<br/>Latin words + CJK bigrams]
  A --> I[Inverted index<br/>field → term → postings<br/>doc, freq, positions]
  Q[query string] --> P[Lexer → parser<br/>term / phrase / prefix / AND OR NOT]
  P --> E[Evaluator<br/>ascending doc arrays<br/>merge-based set ops]
  I --> E
  E --> K[Top-K selection<br/>score desc, id asc]
  K --> H[hits]
  I --> S[Snapshot MSK3<br/>CRC-32 frame]
  I --> W[WAL<br/>torn-tail recovery]
  I --> F[Facets<br/>filter + aggregation]
  B[Browser<br/>WasmGC + js-string] -.-> I

#Three ways to run it

moon build server --target native --release
./_build/native/release/build/server/server.exe --addr 127.0.0.1:7700 --assets console
curl -X POST localhost:7700/index/bulk --data-binary @docs.jsonl
curl "localhost:7700/search?q=parser&facets=license"

for k in 0 1 2; do
  ./server.exe --addr 127.0.0.1:$((7801+k)) --shard-port 127.0.0.1:$((7901+k)) &
done
./server.exe --addr 127.0.0.1:7805 --shards "127.0.0.1:7901,127.0.0.1:7902,127.0.0.1:7903"
# ?dfs=true makes the ranking identical to a single node holding the whole corpus
curl "localhost:7805/search?q=parser&dfs=true"

tools/demo.sh          # starts it and prints the URLs
tools/demo-story.sh    # or: the full story as a script (see docs/demo-video.md)

./server.exe --addr 127.0.0.1:7805 --shards "…:7920,…:7921,…:7922,…:7923"   # after adding the new shard
curl -X POST "localhost:7805/_rebalance?from=3"
curl -X POST "localhost:7805/_repair"      # after a member comes back

#Quick start

let index = @moonseek.Index::new([
  { name: "title", weight: 3.0 },
  { name: "body", weight: 1.0 },
])

index.add({
  id: "doc1",
  fields: [
    ("title", "全文搜索引擎"),
    ("body", "倒排索引是搜索引擎的核心，BM25 负责排序。"),
  ],
})
index.add({
  id: "doc2",
  fields: [("title", "MoonBit"), ("body", "A fast language for WebAssembly")],
})

let hits = index.search("搜索引擎")        // [{ id: "doc1", score: … }]
let hits2 = index.search("title:moonbit")  // field-scoped
let hits3 = index.search("\"memory safety\" OR wasm*")  // phrase + prefix

// Persist (native)
index.save("index.msk")
let restored = @moonseek.Index::load("index.msk")

// Or portable bytes (works on every backend)
let image : Bytes = index.to_bytes()
let restored2 = @moonseek.Index::from_bytes(image)

#CLI

moon run cmd/main -- index ./docs --index index.msk   # index .txt/.md files
moon run cmd/main -- search "搜索引擎" --index index.msk
moon run cmd/main -- search "title:rust" --limit 5
moon run cmd/main -- info --index index.msk

#Browser demo

moon build --target wasm-gc
cp _build/wasm-gc/debug/build/web/web.wasm demo/web.wasm
cd demo && python -m http.server 8000
# open http://localhost:8000 (Chrome / Edge with WasmGC + js-string builtins)

1. Searches the real registry. Every one of the 3,698 packages published on mooncakes.io (22,837 versions) is indexed — package names, author-written keywords, descriptions, and dependency edges. The index is built by the native binary into a 1.37 MB MSK3 snapshot and loaded by the wasm-gc build, which is the strongest evidence that the format is portable. Loading takes ~200 ms; the median query over a 12-query mix is 0.20 ms.
2. Indexes your own files. Drag in .txt / .md / .json / .csv / .log files (or paste text) and they are parsed, tokenized and searchable immediately, with the whole index persistable to IndexedDB as one snapshot blob and reloadable without re-parsing.
3. Shows its own quality. A live benchmark measures query-latency percentiles and the latency-vs-index-size curve in your browser, next to the ablation table from tools/eval: the same corpus and queries under six scoring configurations, so you can see which mechanism is worth how much NDCG.

#Performance

#Architecture

token.mbt      Analyzer: Latin words + CJK bigrams, positions
index.mbt      Inverted index, field stats, BM25, stored fields
query.mbt      Lexer → recursive-descent parser → evaluator
fuzzy.mbt      Levenshtein/Damerau distance, fuzzy search
suggest.mbt    Frequency-weighted did-you-mean
highlight.mbt  Server-side highlight/spans/snippet (codepoint offsets)
facet.mbt      Facet values, filters, aggregations, pagination
bktree.mbt     BK-tree for sub-linear fuzzy lookup
explain.mbt    BM25 breakdown, sort orders, cursor pagination
segment.mbt    SegmentedIndex: immutable segments + tiered merge
wal.mbt        CRC-framed write-ahead log (portable byte layer)
wal_native.mbt Durable add/checkpoint/crash recovery (native)
codec.mbt      Varint / UTF-8 / double / CRC-32 primitives
snapshot.mbt   MSK3 format: versioned payload + checksum + stored fields + facets
store.mbt      Native atomic file persistence (async/fs)
web/           Browser exports (#export_name, js-string builtins)
cmd/main/      CLI: index / search / info
examples/      Runnable examples (basic, persistence)
demo/          Single-page browser demo (60 docs, benchmarks, inspector)

• docs/benchmark.md — performance whitepaper (stress numbers, methodology, limitations)
• docs/comparison.md — feature matrix vs tantivy / Lucene / bleve

#Testing

moon test                    # 299 tests (native)
moon test --target wasm-gc
moon test --target js
moon bench --release         # benchmarks
moon coverage analyze        # coverage report
moon run bench/stress --target native --release   # 20k-doc stress benchmark

#Roadmap

• Segment merging & incremental indexing
• Result highlighting API
• Fuzzy / edit-distance queries
• Did-you-mean spelling correction
• Write-ahead log with crash recovery
• Faceted search & aggregations
• BK-tree accelerated fuzzy matching
• Score explanation, sort orders, cursor pagination
• Merge-based evaluator over ascending doc arrays (no per-query score map)
• Distributed sharded search: consistent hashing, write replication, cross-shard aggregations, failover
• Rebalancing executor and coordinator election (see the list in docs/cluster.md)

#License