#MoonSearch

[!IMPORTANT] MoonSearch source metadata is currently at 0.9.0 and is still evolving. Read Current limitations before adopting it for production data or a long-lived persistence contract.

#Contents

• Highlights
• Installation
• Library quick start
• CLI quick start
• Browser Wasm Demo
• Architecture
• Text analysis
• Chinese analysis
• Queries and scoring
• Highlighting
• Index lifecycle and persistence
• Benchmarks
• Package layout
• Development
• Current limitations
• Roadmap
• Contributing
• License

#Highlights

• MoonBit-native core — the indexing, storage, query, scoring, analysis, highlighting, CLI, and benchmark paths are implemented in MoonBit.
• Typed Schema and Fast Fields — text, keyword, i64/u64/f64, bool, date, and bytes fields independently declare indexed, stored, fast, and cardinality behavior; fast columns are addressable by docID.
• Pluggable analysis — lazy TokenStream, open Tokenizer and TokenFilter traits, composed TextAnalyzer pipelines, and snapshot-based TokenizerManager registries.
• English and Unicode analysis — raw, whitespace, default lowercase, n-gram, stop-word, long-token, and Porter2 English stemming components.
• Optional Chinese analysis — injected dictionaries, longest matching, weighted frequency-DAG routing, optional B/M/E/S HMM recognition, and dictionary-gated search-mode expansion.
• Full-text queries — Term/Boolean/Phrase, prefix/wildcard/regex/fuzzy and range queries, compound scoring, phrase slop/prefix, and graph-aware analysis.
• BM25 and global Top-K — snapshot-wide statistics are shared across immutable Segments before one global result selection.
• Filtering, sorting, and analytics — exact/range/exists filters, deterministic multi-field sorting, Count/TopDocs/MultiCollector, terms/range facets, and numeric/date aggregations.
• Durable index lifecycle — Directory v2 atomic publication and capability detection, two-phase commits, native OS writer locks, Reader reload, atomic updates, integrity checking, recovery/GC, and merge scheduling.
• UTF-8-safe highlighting — persisted postings offsets, structured source byte ranges, query-tree-aware highlighting, and caller-defined markers.
• Practical tooling — a UTF-8 JSONL CLI, machine-readable JSON output, four-target CI, release-mode benchmarks, and a browser WasmGC demo.

#Installation

moon add Lucius646/MoonSearch@0.9.0

///|
import {
  "Lucius646/MoonSearch",
}

#Library quick start

Schema → Documents → SegmentWriter → Segment → QueryParser → Searcher → Top-K

///|
fn main {
  let tokenizers = @MoonSearch.TokenizerManager::with_defaults()

  let builder = @MoonSearch.SchemaBuilder::new()
  let title = builder.add_text_field(
    "title",
    @MoonSearch.TextOptions::new(true, true).with_tokenizer("default"),
  )
  let body = builder.add_text_field(
    "body",
    @MoonSearch.TextOptions::new(true, true).with_tokenizer("en_stem"),
  )
  let schema = builder.build()

  let writer = @MoonSearch.SegmentWriter::with_schema_and_tokenizers(
    schema, tokenizers,
  ) catch {
    error => abort(error.to_string())
  }

  let first = @MoonSearch.Document::new()
  first.add_text(title, "MoonSearch")
  first.add_text(body, "A MoonBit full-text search engine")
  ignore(writer.add_document(first))

  let second = @MoonSearch.Document::new()
  second.add_text(title, "Language guide")
  second.add_text(body, "MoonBit language documentation")
  ignore(writer.add_document(second))

  let searcher = @MoonSearch.Searcher::new(writer.finish())
  let parser = @MoonSearch.QueryParser::new(schema, tokenizers)
  let query = parser.parse_query(body, "SEARCHES") catch {
    error => abort(error.to_string())
  }

  let hits = searcher.search(query, @MoonSearch.TopKCollector::new(10))
  for hit in hits {
    match searcher.doc(hit.address) {
      Some(document) => {
        let titles = document.texts_for(title)
        println("score=\{hit.score} title=\{titles[0]}")
      }
      None => ()
    }
  }
}

moon run cmd/main --frozen

#CLI quick start

moonsearch index    append UTF-8 JSONL documents as one immutable Segment
moonsearch search   search one indexed field
moonsearch inspect  show snapshot and Schema metadata

moon run cmd/moonsearch --frozen -- --help

#1. Prepare JSONL

{"title":"MoonSearch","body":"MoonBit full text search engine"}
{"title":"MoonBit Guide","body":["language documentation","MoonBit basics"]}
{"title":"Search Notes","body":"UTF-8 indexing and highlighting"}

#2. Build an index

moon run cmd/moonsearch --frozen -- index \
  --index .moonsearch-index \
  --input cmd/moonsearch/testdata/documents.jsonl \
  --field title \
  --field body \
  --tokenizer default

#3. Search

moon run cmd/moonsearch --frozen -- search \
  --index .moonsearch-index \
  --field body \
  --query "moonbit search" \
  --operator and \
  --limit 10

moon run cmd/moonsearch --frozen -- search \
  --index .moonsearch-index \
  --field body \
  --query "full text" \
  --phrase \
  --highlight \
  --fragment-size 160 \
  --json

moon run cmd/moonsearch --frozen -- search \
  --index .moonsearch-index \
  --field body \
  --query 'title:MoonSearch OR body:"full text"' \
  --query-string \
  --json

#4. Inspect

moon run cmd/moonsearch --frozen -- inspect \
  --index .moonsearch-index \
  --json

• 0 for success;
• 1 for input, I/O, index, or query failures;
• 2 for invalid arguments.

#Browser Wasm Demo

npm run demo:build
npm run demo:serve

• Term, requiring exactly one analyzed term;
• Boolean · OR and Boolean · AND;
• positional Phrase matching;
• strict Query-string expressions with fields, grouping, Boolean operators, quoted phrases/slop, ranges, wildcard/regex/fuzzy terms, +/-, boosts, and escaping.

npm install
npm run test:data
npm run test:wasm
npm run test:browser
npm run benchmark:browser

#Architecture

Document
   │
   ▼
Schema ──→ TokenizerManager ──→ TokenStream / TokenFilter
   │                                  │
   └──────────────────────────────────┘
                    │
                    ▼
              SegmentWriter
                    │
                    ▼
        immutable Segment + stored fields
                    │
                    ▼
       Directory + generation manifest
                    │
                    ▼
       IndexReader snapshot → Searcher
                    │              │
                    │              ▼
                    │      Query → Weight → Scorer
                    │              │
                    └──────────────┴─→ TopKCollector
                                      │
                                      ▼
                         StoredDocument + Highlighter

core ──→ schema ──→ analysis ──→ index ──→ query ──→ store
                         │
                         └─→ analysis/chinese

internal/codec  shared binary primitives used by index and store
benchmarks      deterministic public-API performance fixtures
cmd/main        embedded lifecycle example
cmd/moonsearch  JSONL command-line application
tests/integration  root-facade cross-package tests

#Text analysis

#Chinese analysis

• longest dictionary matching with mixed-script fallback;
• weighted frequency-DAG route selection;
• optional B/M/E/S Viterbi recognition for unknown Han runs;
• dictionary-gated two- and three-character search-mode subwords;
• separate precise query and flattened index pipelines.

///|
let lexicon = @MoonSearch.ChineseLexicon::from_dictionary_text(
  (
    #|月兔 100
    #|全文 80
    #|搜索 120
    #|搜索引擎 200
    #|中文 100
    #|分词 90
    #|
  ),
) catch {
  error => abort(error.to_string())
}

///|
let index_tokenizers = @MoonSearch.TokenizerManager::with_defaults()
index_tokenizers.register(
  "zh_search",
  @MoonSearch.chinese_dag_search_index_analyzer(lexicon),
) catch {
  error => abort(error.to_string())
}

///|
let query_tokenizers = @MoonSearch.TokenizerManager::with_defaults()
query_tokenizers.register(
  "zh_search",
  @MoonSearch.chinese_dag_analyzer(lexicon),
) catch {
  error => abort(error.to_string())
}

#Queries and scoring

///|
let parser = @MoonSearch.QueryStringParser::new(schema, tokenizers, [
  title, body,
])

///|
let ast = parser.parse_ast("title:MoonBit AND body:\"full text\"") catch {
  error => abort(error.to_string())
}

///|
let query = parser.build_query_from_ast(ast) catch {
  error => abort(error.to_string())
}

///|
let query = @MoonSearch.BooleanQuery::new([
  @MoonSearch.BooleanClause::new(
    @MoonSearch.Occur::Should,
    @MoonSearch.BoostQuery::new(title_query, 3.0),
  ),
  @MoonSearch.BooleanClause::new(
    @MoonSearch.Occur::Should,
    @MoonSearch.BoostQuery::new(body_query, 1.0),
  ),
])

#Highlighting

• text is a valid UTF-8 fragment of the original stored value;
• fragment and match offsets are source-relative UTF-8 byte ranges;
• prefix_omitted and suffix_omitted describe truncation;
• Terms mode highlights normalized term matches;
• Phrase mode highlights only real positional phrase occurrences;
• Token Graph paths and multi-position tokens remain phrase-aware.

///|
let highlighter = @MoonSearch.Highlighter::new(schema, tokenizers)

///|
let fragment = highlighter.highlight(
  body,
  "MOONBIT SEARCH",
  stored_body,
  @MoonSearch.HighlightOptions::new()
  .with_mode(@MoonSearch.HighlightMode::Phrase)
  .with_max_chars(160),
) catch {
  error => abort(error.to_string())
}

///|
match fragment {
  Some(fragment) => println(fragment.render("<mark>", "</mark>"))
  None => ()
}

#Index lifecycle and persistence

SegmentWriter.finish()
        │
        ▼
IndexWriter.commit(segment)
        │
        ▼
published generation ──→ IndexReader snapshot ──→ Searcher

• MemoryDirectory provides an in-process backend for tests and embedded use.
• FsDirectory persists Segment and manifest data beneath an application-owned directory.
• prepare_commit persists and syncs immutable data without changing readers; commit_prepared atomically replaces the visible manifest, and rollback removes an unpublished Segment;
• native filesystem writers use crash-released OS file locks; portable targets expose weaker guarantees through DirectoryCapabilities;
• IndexReader.reload/open_if_changed adopts a later generation while Searchers created before reload retain their previous immutable snapshot;
• check_index validates manifest references, Segment checksums, tombstones, and Schema compatibility while reporting orphan and temporary files;
• MergePolicy and MergeScheduler provide deterministic count thresholds and a hard input-byte budget;
• each IndexReader observes the generation it opened; publishing a later generation does not mutate the existing reader snapshot;
• term deletion records deleted document IDs without rewriting immutable Segment postings;
• Merge copies live index structures into one new Segment and does not re-analyze stored documents;
• stored text preserves multiple values per field and their insertion order;
• Segment v5 stores block postings/positions, skip metadata, front-coded terms, compressed lazy stored fields, typed/Fast Field values, source offsets, position length, and value index;
• the decoder retains compatibility with earlier Segment v1/v2/v3/v4 files.

#Benchmarks

• default mixed-UTF-8, English stemming, and Chinese DAG analysis;
• schema-aware construction of a 256-document Segment;
• Term, Boolean AND, and Phrase Top-K search over 2,048 documents;
• Terms and Phrase highlighting over mixed UTF-8 source text.

moon bench benchmarks \
  --release \
  --target native \
  --deny-warn \
  --frozen \
  --no-parallelize

moon bench benchmarks --build-only --target all --deny-warn --frozen

#Package layout

#Development

• a current MoonBit toolchain;
• Git;
• Node.js 22 and a modern Chromium browser for the Browser Demo tests;
• no external service is required for the test suite, benchmarks, or demo.

# Format MoonBit sources
moon fmt

# Refresh generated public interfaces
moon info

# Check every supported backend and reject warnings
moon check --target all --deny-warn --frozen

# Run the full test suite on every backend
moon test --target all --deny-warn --frozen

# Run the embedded example
moon run cmd/main --frozen

# Build benchmarks on every backend without measuring
moon bench benchmarks --build-only --target all --deny-warn --frozen

# Measure the native release build
moon bench benchmarks --release --target native --deny-warn --frozen --no-parallelize

# Build the WasmGC demo artifact and inspect its ABI
npm run test:wasm

# Execute browser query and load-failure smoke tests
npm run test:browser

• WebAssembly (wasm);
• WebAssembly GC (wasm-gc);
• JavaScript (js);
• native C backend (native).

#Current limitations

• The Schema supports typed fields and Fast Fields, but not dynamic JSON, IP, vector, or geo fields.
• QueryStringParser supports fields, grouping, Boolean operators, ranges, wildcard/regex/fuzzy terms, phrase slop, boosts, escaping, and lenient recovery, but is not a complete syntax clone of every Lucene query feature.
• Highlighting supports re-analysis and Segment v5 postings offsets; query-string highlighting is field-aware and ignores prohibited clauses.
• The CLI loads only built-in tokenizers and cannot directly load application dictionaries, HMM tables, or arbitrary tokenizer plugins.
• Chinese dictionaries and HMM models are injected by the application; no large language resource is bundled.
• Segment v5 reads v1/v2/v3/v4, but remains a project format rather than a Tantivy or Lucene index format.
• The Browser Demo uses fixed title and body fields with one explicit Analyzer Profile and keeps only one replaceable in-memory Worker index. Its 137-term project-authored lexicon demonstrates DAG/search-mode behavior but is not a production Chinese dictionary.
• Benchmark numbers are comparable only across equivalent hardware, runtime, power mode, fixture version, and MoonBit toolchain.
• The API and persistence format may evolve before a stable release.

#Roadmap

• document model, Schema, inverted index, postings, stored text, and BM25;
• immutable multi-segment storage, manifest generations, deletion, and Merge;
• streaming analysis, English stemming, Token Graph queries, and Chinese DAG/HMM/search-mode analysis;
• JSONL CLI, UTF-8-safe highlighting, and deterministic benchmarks;
• M6d Browser Demo with WasmGC execution and structured BM25/highlight results;
• M6e offline corpus Demo with bilingual Wikipedia snapshots, Worker isolation, progress/cancellation, and large-corpus browser baselines.
• M6f bilingual Analyzer Profiles with Chinese DAG/search-mode, English Porter2 stemming, mixed-script processing, and query-token previews.
• M7 strict Query-string v1 with a Schema-independent AST, field/Analyzer binding, UTF-8-spanned errors, CLI integration, and Browser/Wasm execution.
• M8 scalable execution core with an ordered term dictionary, seekable posting cursors and Scorers, bounded Top-K collection, cached snapshot statistics, and Segment v3 delta/varint postings.
• M9 durable index lifecycle with Directory v2, two-phase commit/rollback, native writer locks, Reader reload, atomic updates, recovery, CheckIndex, garbage collection, and merge scheduling.
• M10 typed fields and collectors with Fast Fields, Segment v4, typed filtering/ranges, stable multi-field sorting, facets, and aggregations.
• M11 advanced execution and storage with Segment v5 block postings/skip, front-coded terms, compressed lazy stored fields, safe WAND, memory-budgeted flush, cooperative segment execution plans, and merge backpressure.
• M12 query and analysis completeness with multi-term/compound queries, phrase slop/prefix, scoring explanations, lenient Query-string recovery, persisted-offset highlighting, char filters, synonym graphs, and analyzer resource fingerprints.

#Contributing

1. keep package dependencies acyclic and responsibility-focused;
2. add integration tests for public behavior and white-box tests for local invariants;
3. run moon info and inspect generated interface changes;
4. run formatting, four-target checks, and four-target tests;
5. build the benchmark package when changing a measured path.

#Design references

• Tantivy
• Apache Lucene
• MoonBit

#License