纯 MoonBit 零 FFI 的本地全文检索与 RAG 检索库:中英日分词、BM25 倒排索引、向量索引、RRF 混合检索与 LLM 上下文组装,可编译为 wasm / wasm-gc / js / native 多后端。
# 添加依赖
moon add niuniu513-ask/MoonRetrievelet engine = @lib.Engine::new()
engine.add_document("guide", "MoonBit 是面向云边计算的编程语言。")
engine.add_document("faq", "RAG 是检索增强生成。")
let results = engine.search("RAG 是什么", top_k=3)
let cb = @lib.ContextBuilder::new(max_tokens=500)
println(cb.build("RAG 是什么", results))# 建索引
moon run cmd/main --target native -- index examples/notes demo-index.json
# 检索
moon run cmd/main --target native -- query demo-index.json "MoonBit 黑客松" -k 3
# 生成 LLM 上下文
moon run cmd/main --target native -- context demo-index.json "黑客松奖励" -k 3
# 索引统计
moon run cmd/main --target native -- stats demo-index.json
# 短语检索(可加 --highlight)
moon run cmd/main --target native -- phrase demo-index.json "MoonBit 黑客松" -k 3
# 布尔检索 / 前缀检索
moon run cmd/main --target native -- boolean demo-index.json "MoonBit AND RAG" -k 3
moon run cmd/main --target native -- prefix demo-index.json "moon" -k 3MoonRetrieve.mbt 包说明
tokenizer.mbt 分词器
chunker.mbt 分块器
index.mbt BM25 倒排索引
vector.mbt 向量索引与 RRF 融合
context.mbt LLM 上下文组装
engine.mbt 一站式引擎
evaluation.mbt Precision/Recall/MRR/MAP 等检索评估指标
suggest.mbt Unicode 查询纠错、推荐与前缀补全
query.mbt 必选/排除/短语/前缀结构化查询
result_ops.mbt 融合、去重、分组与多样化
summary.mbt 关键词、摘要与命中片段
cmd/main/ CLI
examples/notes/ 示例文档
docs/ 使用教程、申报书、差异化说明与自查清单moon test # 100 个测试(native / wasm-gc / js 由 CI 覆盖)
moon check
moon build --target wasm-gc
moon bench # 3 项基准(分词 / 建索引 / 检索)moon run examples/quickstart --target wasm-gcfn ContextBuilder::build(self : ContextBuilder, query : String, results : Array[SearchResult]) -> Stringpub struct EvaluationSummary {
query_count : Int
mean_precision : Double
mean_recall : Double
mean_f1 : Double
mean_reciprocal_rank : Double
mean_average_precision : Double
hit_rate : Double
} derive(Debug)fn IndexStats::new(doc_count : Int, term_count : Int, total_tokens : Int, avg_doc_len : Double) -> IndexStatspub struct ResultGroup {
doc_id : String
best_score : Double
total_score : Double
chunk_count : Int
chunks : Array[SearchResult]
} derive(Debug)pub struct RetrievalMetrics {
cutoff : Int
retrieved : Int
relevant : Int
relevant_retrieved : Int
precision : Double
recall : Double
f1 : Double
hit_rate : Double
reciprocal_rank : Double
average_precision : Double
r_precision : Double
} derive(Debug)fn SearchIndex::search_boolean(self : SearchIndex, query : String, top_k? : Int) -> Array[SearchResult]fn SearchIndex::search_phrase(self : SearchIndex, query : String, top_k? : Int) -> Array[SearchResult]fn SearchIndex::search_prefix(self : SearchIndex, prefix : String, top_k? : Int) -> Array[SearchResult]fn diversify_results(results : Array[SearchResult], similarity_threshold? : Double, limit? : Int, tokenizer? : Tokenizer) -> Array[SearchResult]fn edit_distance(left : String, right : String) -> Intfn evaluate_ranking(ranked : Array[String], relevant : Array[String], cutoff? : Int) -> RetrievalMetricsfn evaluate_results(results : Array[SearchResult], relevant : Array[String], cutoff? : Int) -> RetrievalMetricsfn explain_result(query : String, result : SearchResult, tokenizer? : Tokenizer) -> ResultExplanationfn explain_results(query : String, results : Array[SearchResult], tokenizer? : Tokenizer) -> Array[ResultExplanation]fn filter_results(results : Array[SearchResult], min_score? : Double, max_results? : Int) -> Array[SearchResult]fn filter_with_query_plan(results : Array[SearchResult], plan : QueryPlan, tokenizer? : Tokenizer) -> Array[SearchResult]fn paginate_results(results : Array[SearchResult], offset? : Int, limit? : Int) -> Array[SearchResult]fn result_snippet(text : String, query : String, max_chars? : Int, tokenizer? : Tokenizer) -> Stringfn shared_prefix_length(left : String, right : String) -> Intfn snippets_for_results(results : Array[SearchResult], query : String, max_chars? : Int, tokenizer? : Tokenizer) -> Array[SearchResult]fn suggest_terms(query : String, vocabulary : Array[String], max_distance? : Int, limit? : Int) -> Array[QuerySuggestion]fn summarize_text(text : String, query? : String, max_sentences? : Int, tokenizer? : Tokenizer) -> TextSummaryfn transposition_distance(left : String, right : String) -> Intfn weighted_fuse_results(left : Array[SearchResult], right : Array[SearchResult], left_weight? : Double, right_weight? : Double, limit? : Int) -> Array[SearchResult]Install
Download zip纯 MoonBit 零 FFI 的本地全文检索与 RAG 检索库:中英日分词、BM25 倒排索引、向量索引、RRF 混合检索与 LLM 上下文组装,可编译为 wasm / wasm-gc / js / native 多后端。