🥮 mooncakes.io
    Sign in

    MoonRetrieve

    纯 MoonBit 零 FFI 的本地全文检索与 RAG 检索库：中英日分词、BM25 倒排索引、向量索引、RRF 混合检索与 LLM 上下文组装，可编译为 wasm / wasm-gc / js / native 多后端。

    moonbit
    search
    bm25
    rag
    tokenizer
    wasm
    local-first
    hybrid
    rrf
    Download zip
    Version
    0.6.0
    License
    Apache-2.0
    Repository
    Last updated
    22 days ago
    Downloads
    23

    Dependencies

    #MoonRetrieve

    mooncakes.io CI

    纯 MoonBit 零 FFI 的本地全文检索与 RAG（检索增强生成）检索库。文档在本地完成分词、分块、索引、检索和 LLM 上下文组装，数据不出本地；核心库可编译到 wasm / wasm-gc / js / native 多后端，适合浏览器、边缘设备与隐私敏感场景。

    #与现有 MoonBit 检索项目的差异

    • 不重复造“搜索引擎内核”：区别于 Lucius646/MoonSearch（嵌入式全文检索内核，多 Segment 持久化、Phrase/Boolean 查询），MoonRetrieve 聚焦轻量索引 + 混合检索 + RAG 上下文组装，提供 Engine 一站式 API 和开箱即用的 CLI。
    • 纯库、零 FFI：区别于 houjie/rag-mbt（RAG 管线，native 侧依赖 Python/bge 嵌入与 FFI），MoonRetrieve 核心库无任何 FFI，同一套代码跑 wasm-gc / js / native，适合浏览器与受限边缘环境。
    • 内置多语言分词：中/英/日文，中文单字 + bigram（单字查询也能命中），英文停用词与可选词干化，不依赖外部分词服务。
    • 混合检索 + 上下文组装：BM25 倒排索引与余弦向量索引双路检索，RRF 融合排序；ContextBuilder 按 token 预算把命中片段拼成带来源编号的 LLM 提示词。

    #功能

    • 分词器 Tokenizer：英文/中文/日文、N-gram、停用词、词干化
    • 分块器 Chunker：按段落切分、块间重叠、超长段落硬切
    • BM25 索引 SearchIndex：增量建索引、Top-K 检索、JSON 持久化
    • 倒排驱动 BM25：只为命中文档计分，适合本地较大文档集；2,000 文档基准与复现方法
    • 短语检索 search_phrase：查询短语按原文顺序连续出现
    • 布尔检索 search_boolean：AND / OR / -term 排除
    • 前缀检索 search_prefix：按词条前缀匹配
    • 向量索引 VectorIndex：余弦相似度检索
    • 混合融合 rrf_fuse：Reciprocal Rank Fusion
    • 上下文组装 ContextBuilder：来源去重、分块预算与带引用清单的 ContextReport
    • 文档级检索评测：标注查询集、逐题排名与 Precision/Recall/MRR/MAP 汇总
    • JSONL 批量导入：保留外部文档 ID 与标题，接入已有知识库语料
    • 检索诊断轨迹 Engine::trace：输出排名、来源、分数、命中/缺失词和原始证据文本
    • 结果高亮 highlight：查询词自动标记 **term**
    • 文档删除与统计：SearchIndex::remove / stats、Engine::remove_document / stats
    • 一站式引擎 Engine：文档 → 分块 → 索引 → 检索，支持文档替换和 RAG 上下文构建
    • CLI：index / query / context / evaluate / trace / stats / phrase / boolean / prefix
    • 结果诊断 CLI：snippet 输出限定长度命中片段，explain 输出匹配词和覆盖情况

    完整的科研摘要检索场景与 300 题公开人工标注评测见 SciFact 复现说明。本期 Linux native 实测 Recall@10 为 0.7533、MRR@10 为 0.5847；流程覆盖导入、查询、引用核对和错误分析。语料需从原始发布方下载，不随本仓库分发。

    #快速开始

    环境要求：MoonBit 0.1.20260920+；native 目标需要 C 编译器（Windows 用 MSVC）。

    # 添加依赖
moon add niuniu513-ask/MoonRetrieve

    库用法：

    let engine = @lib.Engine::new()
    engine.add_document("guide", "MoonBit 是面向云边计算的编程语言。")
    engine.add_document("faq", "RAG 是检索增强生成。")

    let results = engine.search("RAG 是什么", top_k=3)
    let cb = @lib.ContextBuilder::new(max_tokens=500)
    println(cb.build("RAG 是什么", results))

    CLI 用法（native 目标）：

    # 建索引
moon run cmd/main --target native -- index examples/notes demo-index.json

# 导入带稳定 ID 的 JSONL（每行包含 id 或 _id、text，可选 title）
moon run cmd/main --target native -- index-jsonl documents.jsonl demo-index.json

# 检索
moon run cmd/main --target native -- query demo-index.json "MoonBit 黑客松" -k 3

# 生成 LLM 上下文
moon run cmd/main --target native -- context demo-index.json "黑客松奖励" -k 3

# 输出可供应用读取的提示词、引用清单与预算信息
moon run cmd/main --target native -- context demo-index.json "MoonBit" -k 3 --tokens 2000 --diverse --json

# 用标注查询集评测文档级检索，JSON 含每题排名与汇总指标
moon run cmd/main --target native -- evaluate demo-index.json examples/evaluation-cases.json -k 3 --json

# 检查一次查询的排名和词项匹配证据
moon run cmd/main --target native -- trace demo-index.json "MoonBit RAG" -k 3 --json

# 索引统计
moon run cmd/main --target native -- stats demo-index.json

# 短语检索（可加 --highlight）
moon run cmd/main --target native -- phrase demo-index.json "MoonBit 黑客松" -k 3

# 布尔检索 / 前缀检索
moon run cmd/main --target native -- boolean demo-index.json "MoonBit AND RAG" -k 3
moon run cmd/main --target native -- prefix demo-index.json "moon" -k 3

    #项目结构

    MoonRetrieve.mbt      包说明
jsonl.mbt            带稳定 ID 的文档批量导入校验
tokenizer.mbt        分词器
chunker.mbt          分块器
index.mbt            BM25 倒排索引
vector.mbt           向量索引与 RRF 融合
context.mbt          LLM 上下文组装
engine.mbt           一站式引擎
evaluation.mbt       Precision/Recall/MRR/MAP 等检索评估指标
trace.mbt            可导出的检索诊断轨迹
suggest.mbt          Unicode 查询纠错、推荐与前缀补全
query.mbt            必选/排除/短语/前缀结构化查询
result_ops.mbt       融合、去重、分组与多样化
summary.mbt          关键词、摘要与命中片段
cmd/main/            CLI
examples/notes/      示例文档
examples/evaluation-cases.json  小型检索评测样例
docs/                使用教程、申报书、差异化说明与自查清单

    #测试与构建

    moon test        # native / wasm-gc / js 由 CI 覆盖
moon check
moon build --target wasm-gc
moon bench       # 5 项基准（分词 / 建索引 / 小集合与 2,000 文档检索）

    #最小示例

    moon run examples/quickstart --target wasm-gc

    #许可证

    Apache-2.0

    #来源与合规

    项目为原创 MoonBit 实现，未复制第三方搜索引擎、RAG 框架、分词器或向量数据库源码。实现参考的公开信息检索概念、示例数据和差异化说明见 docs/SOURCES.md 与 docs/differentiation.md。

    #参赛与维护

    OSC 2026 申请人、仓库账号及历史 Git 作者身份说明见 docs/PARTICIPATION.md。

    #本期实质新增工作

    本期新增的检索评估、查询解析、相关性解释、纠错和结果处理代码分别位于 evaluation.mbt、query.mbt、explain.mbt、suggest.mbt、result_ops.mbt。文档更新时可直接替换旧分块；RAG 上下文可按原始文档去重，并返回实际使用的引用清单和预算用量，应用无需再从提示词中解析来源。

    普通检索与前缀检索现按倒排表计分。在同一 wasm-gc 环境的 2,000 文档基准中，稀有词查询由约 62.8 µs 降至 0.45 µs，常见词由约 1.15 ms 降至 99 µs；负载和复现命令见 性能记录。本期再增加 SciFact 测试集的完整文档级评测，原来的三题样例仅保留作冒烟测试。JSONL 导入保留外部语料 ID，检索诊断和引用报告可以回溯到原文。CI 在 native、wasm-gc 和 js 上运行检查与测试，另设 SciFact 全量评测任务。mooncakes 的发布状态以包页面为准。

    九月黑客松的规则、上一期晋级项目的可借鉴做法及尚未完成的工作见 九月赛对照记录。

    Chunk

    source
    pub struct Chunk {
      doc_id : String
      index : Int
      text : String
    } derive(Eq, ToJson, 
    Debug    , 
    FromJson    )

    文档分块结果。

    Chunk::equal

    source
    fn Chunk::equal(Chunk, Chunk) -> Bool

    Chunk::from_json

    source

    Chunk::new

    source
    fn Chunk::new(doc_id : String, index : Int, text : String) -> Chunk

    Chunk::not_equal

    source
    fn Chunk::not_equal(x : Chunk, y : Chunk) -> Bool

    Chunk::to_json

    source
    fn Chunk::to_json(Chunk) -> Json

    Chunk::to_repr

    source

    Chunker

    source
    pub struct Chunker {
      max_chars : Int
      overlap : Int
    } derive(ToJson, 
    FromJson    )

    文档分块器：按段落切分，块之间保留重叠内容。

    Chunker::chunk

    source
    fn Chunker::chunk(self : Chunker, doc_id : String, text : String) -> Array[Chunk]

    将文档切成带编号的分块。

    Chunker::from_json

    source

    Chunker::new

    source
    fn Chunker::new(max_chars? : Int, overlap? : Int) -> Chunker

    Chunker::to_json

    source
    fn Chunker::to_json(Chunker) -> Json

    Keep derived trait methods available as explicit methods on public types.

    ContextBuilder

    source
    pub struct ContextBuilder {
      max_tokens : Int
      tokenizer : Tokenizer
    } derive(ToJson, 
    FromJson    )

    LLM 上下文组装器：把检索结果按 token 预算拼成可直接喂给大模型的提示词。

    ContextBuilder::build

    source
    fn ContextBuilder::build(self : ContextBuilder, query : String, results : Array[SearchResult]) -> String

    组装上下文：按输入顺序选取预算内的片段，并跳过放不下的结果。

    ContextBuilder::build_diverse

    source
    fn ContextBuilder::build_diverse(self : ContextBuilder, query : String, results : Array[SearchResult], max_sources? : Int) -> String

    每个原始文档只保留最高分分块，再按 token 预算组装上下文。

    ContextBuilder::build_diverse_report

    source
    fn ContextBuilder::build_diverse_report(self : ContextBuilder, query : String, results : Array[SearchResult], max_sources? : Int) -> ContextReport

    Keep the strongest chunk per source and return its citation report.

    ContextBuilder::build_report

    source
    fn ContextBuilder::build_report(self : ContextBuilder, query : String, results : Array[SearchResult]) -> ContextReport

    Select chunks under the retrieved-content budget and retain citation data.

    ContextBuilder::from_json

    source

    ContextBuilder::new

    source
    fn ContextBuilder::new(max_tokens? : Int, tokenizer? : Tokenizer) -> ContextBuilder

    ContextBuilder::to_json

    source

    ContextCitation

    source
    pub struct ContextCitation {
      number : Int
      doc_id : String
      score : Double
      text : String
      estimated_tokens : Int
    } derive(ToJson)

    A selected chunk and its citation number in the generated prompt.

    ContextCitation::to_json

    source

    ContextReport

    source
    pub struct ContextReport {
      prompt : String
      citations : Array[ContextCitation]
      used_content_tokens : Int
      estimated_prompt_tokens : Int
      skipped_results : Int
    } derive(ToJson)

    Prompt plus the exact chunks used to construct it.

    ContextReport::to_json

    source

    Engine

    source
    pub struct Engine {
      chunker : Chunker
      index : SearchIndex
    } derive(ToJson, 
    FromJson    )

    一站式本地检索引擎：文档 -> 分块 -> BM25 索引 -> Top-K 检索。

    Engine::add_document

    source
    fn Engine::add_document(self : Engine, doc_id : String, text : String) -> Unit

    添加一篇文档（自动分块后逐块建索引），返回更新后的引擎。

    Engine::add_documents

    source
    fn Engine::add_documents(self : Engine, docs : Array[(String, String)]) -> Unit

    批量添加文档。

    Engine::build_context

    source
    fn Engine::build_context(self : Engine, query : String, top_k? : Int, max_tokens? : Int) -> String

    检索并按 token 预算生成可直接交给 LLM 的上下文。

    Engine::build_context_report

    source
    fn Engine::build_context_report(self : Engine, query : String, top_k? : Int, max_tokens? : Int) -> ContextReport

    Return the prompt and the exact chunks selected for citation.

    Engine::build_diverse_context

    source
    fn Engine::build_diverse_context(self : Engine, query : String, top_k? : Int, max_tokens? : Int) -> String

    检索后按来源去重，生成覆盖更多文档的 RAG 上下文。

    Engine::build_diverse_context_report

    source
    fn Engine::build_diverse_context_report(self : Engine, query : String, top_k? : Int, max_tokens? : Int) -> ContextReport

    Build a citation report with at most one chunk from each source document.

    Engine::count

    source
    fn Engine::count(self : Engine) -> Int

    索引中的分块总数。

    Engine::evaluate_queries

    source
    fn Engine::evaluate_queries(self : Engine, cases : Array[EvaluationCase], cutoff? : Int) -> EvaluationReport

    Evaluate ranked original documents against a labeled query set.

    Engine::from_json

    source

    Engine::highlight

    source
    fn Engine::highlight(self : Engine, text : String, query : String) -> String

    对文本执行查询词高亮。

    Engine::new

    source
    fn Engine::new(chunker? : Chunker, index? : SearchIndex) -> Engine

    Engine::remove_document

    source
    fn Engine::remove_document(self : Engine, doc_id : String) -> Int

    按原始文档 ID 删除一篇文档及其全部分块，返回删除的分块数。

    Engine::replace_document

    source
    fn Engine::replace_document(self : Engine, doc_id : String, text : String) -> Int

    替换同一原始文档 ID 的全部旧分块，并写入新内容。 返回被删除的旧分块数量。

    Engine::save

    source
    fn Engine::save(self : Engine) -> String

    序列化为 JSON 字符串。

    Engine::search

    source
    fn Engine::search(self : Engine, query : String, top_k? : Int) -> Array[SearchResult]

    检索 Top-K 分块。

    Engine::search_boolean

    source
    fn Engine::search_boolean(self : Engine, query : String, top_k? : Int) -> Array[SearchResult]

    布尔检索 Top-K 分块（AND / OR / -term）。

    Engine::search_explained

    source
    fn Engine::search_explained(self : Engine, query : String, top_k? : Int) -> Array[ResultExplanation]

    检索并返回每条结果的相关性解释。

    Engine::search_phrase

    source
    fn Engine::search_phrase(self : Engine, query : String, top_k? : Int) -> Array[SearchResult]

    短语检索 Top-K 分块。

    Engine::search_prefix

    source
    fn Engine::search_prefix(self : Engine, prefix : String, top_k? : Int) -> Array[SearchResult]

    前缀检索 Top-K 分块。

    Engine::search_snippets

    source
    fn Engine::search_snippets(self : Engine, query : String, top_k? : Int, max_chars? : Int) -> Array[SearchResult]

    检索并返回限定长度的命中片段，适合列表或终端展示。

    Engine::search_sources

    source
    fn Engine::search_sources(self : Engine, query : String, top_k? : Int) -> Array[SearchResult]

    Return the best matching chunk from each original document.

    Engine::stats

    source
    fn Engine::stats(self : Engine) -> IndexStats

    返回索引统计信息。

    Engine::to_json

    source
    fn Engine::to_json(Engine) -> Json

    Engine::trace

    source
    fn Engine::trace(self : Engine, query : String, top_k? : Int) -> RetrievalTrace

    Record ranked chunks and their matching evidence.

    Engine::try_load

    source
    fn Engine::try_load(json : String) -> Result[Engine, String]

    从 JSON 字符串恢复引擎。

    EvaluatedQuery

    source
    pub struct EvaluatedQuery {
      query : String
      ranked_doc_ids : Array[String]
      metrics : RetrievalMetrics
    } derive(ToJson)

    EvaluatedQuery::to_json

    source

    EvaluationCase

    source
    pub struct EvaluationCase {
      query : String
      relevant_doc_ids : Array[String]
    } derive(ToJson, 
    FromJson    )

    One query with relevant original document IDs, not chunk IDs.

    EvaluationCase::from_json

    source

    EvaluationCase::new

    source
    fn EvaluationCase::new(query : String, relevant_doc_ids : Array[String]) -> EvaluationCase

    EvaluationCase::to_json

    source

    EvaluationReport

    source
    pub struct EvaluationReport {
      cutoff : Int
      queries : Array[EvaluatedQuery]
      summary : EvaluationSummary
    } derive(ToJson)

    EvaluationReport::to_json

    source

    EvaluationSummary

    source
    pub struct EvaluationSummary {
      query_count : Int
      mean_precision : Double
      mean_recall : Double
      mean_f1 : Double
      mean_reciprocal_rank : Double
      mean_average_precision : Double
      hit_rate : Double
    } derive(ToJson, 
    Debug    )

    Aggregate metrics over multiple queries.

    EvaluationSummary::to_json

    source

    EvaluationSummary::to_repr

    source

    IndexStats

    source
    pub struct IndexStats {
      doc_count : Int
      term_count : Int
      total_tokens : Int
      avg_doc_len : Double
    } derive(ToJson, 
    Debug    , 
    FromJson    )

    索引统计信息。

    IndexStats::from_json

    source

    IndexStats::new

    source
    fn IndexStats::new(doc_count : Int, term_count : Int, total_tokens : Int, avg_doc_len : Double) -> IndexStats

    IndexStats::to_json

    source
    fn IndexStats::to_json(IndexStats) -> Json

    IndexStats::to_repr

    source

    Keyword

    source
    pub struct Keyword {
      term : String
      count : Int
      frequency : Double
    } derive(Eq, 
    Debug    )

    One extracted keyword and its corpus-local frequency.

    Keyword::equal

    source
    fn Keyword::equal(Keyword, Keyword) -> Bool

    Keyword::not_equal

    source
    fn Keyword::not_equal(x : Keyword, y : Keyword) -> Bool

    Keyword::to_repr

    source

    Posting

    source
    pub struct Posting {
      doc_id : Int
      tf : Int
    } derive(Eq, ToJson, 
    Debug    , 
    FromJson    )

    倒排索引中的一条 posting：文档编号 + 词频。

    Posting::equal

    source
    fn Posting::equal(Posting, Posting) -> Bool

    Posting::from_json

    source

    Posting::not_equal

    source
    fn Posting::not_equal(x : Posting, y : Posting) -> Bool

    Posting::to_json

    source
    fn Posting::to_json(Posting) -> Json

    Posting::to_repr

    source

    QueryClause

    source
    pub struct QueryClause {
      value : String
      field : String?
      required : Bool
      excluded : Bool
      phrase : Bool
      prefix : Bool
    } derive(Eq, 
    Debug    )

    One parsed query clause.

    QueryClause::equal

    source
    fn QueryClause::equal(QueryClause, QueryClause) -> Bool

    QueryClause::not_equal

    source
    fn QueryClause::not_equal(x : QueryClause, y : QueryClause) -> Bool

    QueryClause::to_repr

    source

    QueryPlan

    source
    pub struct QueryPlan {
      original : String
      clauses : Array[QueryClause]
    } derive(Eq, 
    Debug    )

    Parsed query language representation.

    QueryPlan::equal

    source
    fn QueryPlan::equal(QueryPlan, QueryPlan) -> Bool

    QueryPlan::excluded_values

    source
    fn QueryPlan::excluded_values(self : QueryPlan) -> Array[String]

    QueryPlan::has_field

    source
    fn QueryPlan::has_field(self : QueryPlan, field : String) -> Bool

    QueryPlan::matches

    source
    fn QueryPlan::matches(self : QueryPlan, text : String, tokenizer? : Tokenizer) -> Bool

    Test required and excluded clauses against plain text. Optional clauses do not filter the document; they are used for ranking.

    QueryPlan::not_equal

    source
    fn QueryPlan::not_equal(x : QueryPlan, y : QueryPlan) -> Bool

    QueryPlan::positive_values

    source
    fn QueryPlan::positive_values(self : QueryPlan) -> Array[String]

    QueryPlan::required_values

    source
    fn QueryPlan::required_values(self : QueryPlan) -> Array[String]

    QueryPlan::search_text

    source
    fn QueryPlan::search_text(self : QueryPlan) -> String

    QueryPlan::to_repr

    source

    QuerySuggestion

    source
    pub struct QuerySuggestion {
      value : String
      distance : Int
      similarity : Double
    } derive(Eq, 
    Debug    )

    One spelling or completion suggestion.

    QuerySuggestion::equal

    source

    QuerySuggestion::not_equal

    source
    fn QuerySuggestion::not_equal(x : QuerySuggestion, y : QuerySuggestion) -> Bool

    QuerySuggestion::to_repr

    source

    ResultExplanation

    source
    pub struct ResultExplanation {
      doc_id : String
      score : Double
      matched_terms : Array[String]
      missing_terms : Array[String]
      matched_required : Int
      total_required : Int
      token_overlap : Double
    } derive(
    Debug    )

    Human-readable relevance evidence for one search result.

    ResultExplanation::to_repr

    source

    ResultGroup

    source
    pub struct ResultGroup {
      doc_id : String
      best_score : Double
      total_score : Double
      chunk_count : Int
      chunks : Array[SearchResult]
    } derive(
    Debug    )

    Aggregated result information for chunks from the same source document.

    ResultGroup::to_repr

    source

    RetrievalMetrics

    source
    pub struct RetrievalMetrics {
      cutoff : Int
      retrieved : Int
      relevant : Int
      relevant_retrieved : Int
      precision : Double
      recall : Double
      f1 : Double
      hit_rate : Double
      reciprocal_rank : Double
      average_precision : Double
      r_precision : Double
    } derive(ToJson, 
    Debug    )

    Metrics for one ranked retrieval result list.

    RetrievalMetrics::to_json

    source

    RetrievalMetrics::to_repr

    source

    RetrievalTrace

    source
    pub struct RetrievalTrace {
      query : String
      query_terms : Array[String]
      top_k : Int
      hits : Array[TraceHit]
      source_count : Int
    } derive(ToJson)

    A JSON-serializable record of one retrieval run.

    RetrievalTrace::to_json

    source

    ScoredDoc

    source
    pub struct ScoredDoc {
      id : String
      score : Double
    } derive(Eq, ToJson, 
    Debug    , 
    FromJson    )

    向量检索或混合检索的排序结果。

    ScoredDoc::equal

    source
    fn ScoredDoc::equal(ScoredDoc, ScoredDoc) -> Bool

    ScoredDoc::from_json

    source

    ScoredDoc::new

    source
    fn ScoredDoc::new(id : String, score : Double) -> ScoredDoc

    ScoredDoc::not_equal

    source
    fn ScoredDoc::not_equal(x : ScoredDoc, y : ScoredDoc) -> Bool

    ScoredDoc::to_json

    source
    fn ScoredDoc::to_json(ScoredDoc) -> Json

    ScoredDoc::to_repr

    source

    SearchIndex

    source
    pub struct SearchIndex {
      ids : Array[String]
      texts : Array[String]
      lens : Array[Int]
      postings : Map[String, Array[Posting]]
      total_docs : Int
      sum_len : Int
      avg_dl : Double
      k1 : Double
      b : Double
      tokenizer : Tokenizer
    } derive(ToJson, 
    FromJson    )

    BM25 倒排索引。

    支持增量添加文档、Top-K 检索，以及 JSON 序列化/反序列化。 设计为不可变更新风格：add 返回添加后的新索引。

    SearchIndex::add

    source
    fn SearchIndex::add(self : SearchIndex, id : String, text : String) -> Unit

    向索引添加一篇文档，返回新索引。id 可以是文档路径或分块 ID。

    SearchIndex::count

    source
    fn SearchIndex::count(self : SearchIndex) -> Int

    当前索引中的文档/分块数量。

    SearchIndex::from_json

    source

    SearchIndex::new

    source
    fn SearchIndex::new(k1? : Double, b? : Double, tokenizer? : Tokenizer) -> SearchIndex

    SearchIndex::remove

    source
    fn SearchIndex::remove(self : SearchIndex, id : String) -> Bool

    按文档 ID 删除一篇文档，返回是否删除成功。

    SearchIndex::remove_if

    source
    fn SearchIndex::remove_if(self : SearchIndex, pred : (String) -> Bool) -> Int

    按条件删除文档，返回删除数量。删除后内部编号重建，复杂度 O(N)。

    SearchIndex::save

    source
    fn SearchIndex::save(self : SearchIndex) -> String

    序列化为 JSON 字符串。

    SearchIndex::search

    source
    fn SearchIndex::search(self : SearchIndex, query : String, top_k? : Int) -> Array[SearchResult]

    对查询执行 BM25 检索，返回按分数降序排列的 Top-K 结果。

    SearchIndex::search_boolean

    source
    fn SearchIndex::search_boolean(self : SearchIndex, query : String, top_k? : Int) -> Array[SearchResult]

    布尔检索：支持 AND / OR 连接词与 -term 排除。

    示例：
    • "moonbit AND rag"：同时包含 moonbit 与 rag
    • "moonbit OR rag"：包含任一即可
    • "moonbit -old"：包含 moonbit 且不含 old

    SearchIndex::search_phrase

    source
    fn SearchIndex::search_phrase(self : SearchIndex, query : String, top_k? : Int) -> Array[SearchResult]

    短语检索：查询短语必须按原文顺序连续出现，结果仍按 BM25 分数排序。

    SearchIndex::search_prefix

    source
    fn SearchIndex::search_prefix(self : SearchIndex, prefix : String, top_k? : Int) -> Array[SearchResult]

    前缀检索：匹配以给定前缀开头的词条，结果按 BM25 分数排序。

    SearchIndex::stats

    source
    fn SearchIndex::stats(self : SearchIndex) -> IndexStats

    返回索引统计信息。

    SearchIndex::to_json

    source
    fn SearchIndex::to_json(SearchIndex) -> Json

    SearchIndex::try_load

    source
    fn SearchIndex::try_load(json : String) -> Result[SearchIndex, String]

    从 JSON 字符串恢复索引。

    SearchResult

    source
    pub struct SearchResult {
      doc_id : String
      score : Double
      text : String
    } derive(Eq, ToJson, 
    Debug    , 
    FromJson    )

    检索结果：命中的分块/文档及其 BM25 分数。

    SearchResult::equal

    source

    SearchResult::from_json

    source

    SearchResult::new

    source
    fn SearchResult::new(doc_id : String, score : Double, text : String) -> SearchResult

    SearchResult::not_equal

    source
    fn SearchResult::not_equal(x : SearchResult, y : SearchResult) -> Bool

    SearchResult::to_json

    source

    SearchResult::to_repr

    source

    TextSummary

    source
    pub struct TextSummary {
      text : String
      original_sentences : Int
      selected_sentences : Int
      estimated_tokens : Int
    } derive(Eq, 
    Debug    )

    Query-focused extractive summary.

    TextSummary::equal

    source
    fn TextSummary::equal(TextSummary, TextSummary) -> Bool

    TextSummary::not_equal

    source
    fn TextSummary::not_equal(x : TextSummary, y : TextSummary) -> Bool

    TextSummary::to_repr

    source

    Tokenizer

    source
    pub struct Tokenizer {
      min_token_len : Int
      bigram_cjk : Bool
      stem_english : Bool
      stopwords : Array[String]
    } derive(ToJson, 
    FromJson    )

    分词器配置。

    默认行为：
    • 英文/数字按字母数字串切分并转小写，过滤长度小于 min_token_len 的词
    • 中日韩文字按单字 + 相邻二字 bigram 切分
    • 可配置停用词与英文词干化

    Tokenizer::estimate_tokens

    source
    fn Tokenizer::estimate_tokens(self : Tokenizer, text : String) -> Int

    估算文本的 token 数（按当前分词结果计数，供上下文预算使用）。

    Tokenizer::from_json

    source

    Tokenizer::new

    source
    fn Tokenizer::new(min_token_len? : Int, bigram_cjk? : Bool, stem_english? : Bool, stopwords? : Array[String]) -> Tokenizer

    Tokenizer::to_json

    source
    fn Tokenizer::to_json(Tokenizer) -> Json

    Tokenizer::tokenize

    source
    fn Tokenizer::tokenize(self : Tokenizer, text : String) -> Array[String]

    对文本分词，返回词条数组。

    TraceHit

    source
    pub struct TraceHit {
      rank : Int
      doc_id : String
      source_doc_id : String
      score : Double
      matched_terms : Array[String]
      missing_terms : Array[String]
      token_overlap : Double
      text : String
    } derive(ToJson)

    One ranked hit and the terms supporting its retrieval score.

    TraceHit::to_json

    source
    fn TraceHit::to_json(TraceHit) -> Json

    VectorIndex

    source
    pub struct VectorIndex {
      items : Array[ScoredDoc]
      vectors : Array[Array[Double]]
    } derive(ToJson, 
    FromJson    )

    稠密向量索引（暴力检索，适合中小规模本地 RAG）。

    VectorIndex::add

    source
    fn VectorIndex::add(self : VectorIndex, id : String, vector : Array[Double]) -> Unit

    添加一个带向量的文档/分块，返回更新后的索引。

    VectorIndex::count

    source
    fn VectorIndex::count(self : VectorIndex) -> Int

    向量数量。

    VectorIndex::from_json

    source

    VectorIndex::new

    source

    VectorIndex::search

    source
    fn VectorIndex::search(self : VectorIndex, query : Array[Double], top_k? : Int) -> Array[ScoredDoc]

    按余弦相似度检索 Top-K。

    VectorIndex::to_json

    source
    fn VectorIndex::to_json(VectorIndex) -> Json

    average_precision_at

    source
    fn average_precision_at(ranked : Array[String], relevant : Array[String], cutoff : Int) -> Double

    complete_prefix

    source
    fn complete_prefix(prefix : String, vocabulary : Array[String], limit? : Int) -> Array[String]

    Prefix completions ranked by shortest completion and lexical order.

    corpus_coverage

    source
    fn corpus_coverage(rankings : Array[Array[String]], corpus_ids : Array[String], cutoff? : Int) -> Double

    correct_query

    source
    fn correct_query(query : String, vocabulary : Array[String], max_distance? : Int) -> String

    Replace each whitespace-delimited query word with its nearest vocabulary entry when the distance threshold is met.

    cosine

    source
    fn cosine(a : Array[Double], b : Array[Double]) -> Double

    计算两个向量的余弦相似度。维度不一致或零向量返回 0。

    deduplicate_by_source

    source
    fn deduplicate_by_source(results : Array[SearchResult], limit? : Int) -> Array[SearchResult]

    Keep only the highest-scored chunk for each source document.

    default_stopwords

    source
    fn default_stopwords() -> Array[String]

    常用中文停用词。

    diversify_results

    source
    fn diversify_results(results : Array[SearchResult], similarity_threshold? : Double, limit? : Int, tokenizer? : Tokenizer) -> Array[SearchResult]

    Greedy diversity filter. Results above similarity_threshold with an already-selected result are skipped.

    edit_distance

    source
    fn edit_distance(left : String, right : String) -> Int

    Unicode-aware Levenshtein edit distance.

    edit_similarity

    source
    fn edit_similarity(left : String, right : String) -> Double

    evaluate_ranking

    source
    fn evaluate_ranking(ranked : Array[String], relevant : Array[String], cutoff? : Int) -> RetrievalMetrics

    evaluate_results

    source
    fn evaluate_results(results : Array[SearchResult], relevant : Array[String], cutoff? : Int) -> RetrievalMetrics

    explain_result

    source
    fn explain_result(query : String, result : SearchResult, tokenizer? : Tokenizer) -> ResultExplanation

    explain_results

    source
    fn explain_results(query : String, results : Array[SearchResult], tokenizer? : Tokenizer) -> Array[ResultExplanation]

    explanation_summary

    source
    fn explanation_summary(explanation : ResultExplanation) -> String

    extract_keywords

    source
    fn extract_keywords(text : String, limit? : Int, tokenizer? : Tokenizer) -> Array[Keyword]

    f1_score

    source
    fn f1_score(precision : Double, recall : Double) -> Double

    filter_results

    source
    fn filter_results(results : Array[SearchResult], min_score? : Double, max_results? : Int) -> Array[SearchResult]

    filter_with_query_plan

    source
    fn filter_with_query_plan(results : Array[SearchResult], plan : QueryPlan, tokenizer? : Tokenizer) -> Array[SearchResult]

    format_query_plan

    source
    fn format_query_plan(plan : QueryPlan) -> String

    group_results_by_source

    source
    fn group_results_by_source(results : Array[SearchResult]) -> Array[ResultGroup]

    highlight

    source
    fn highlight(text : String, query : String, tokenizer? : Tokenizer) -> String

    在文本中用 **term** 标记查询命中的词，返回高亮后的文本。

    长词优先、区间不重叠，避免中文单字与 bigram 同时命中时重复标记。

    mean_average_precision

    source
    fn mean_average_precision(rankings : Array[Array[String]], relevance : Array[Array[String]], cutoff? : Int) -> Double

    mean_reciprocal_rank

    source
    fn mean_reciprocal_rank(rankings : Array[Array[String]], relevance : Array[Array[String]], cutoff? : Int) -> Double

    normalize_result_scores

    source
    fn normalize_result_scores(results : Array[SearchResult]) -> Array[SearchResult]

    Min-max normalize scores into [0, 1]. Equal nonzero scores become 1.

    paginate_results

    source
    fn paginate_results(results : Array[SearchResult], offset? : Int, limit? : Int) -> Array[SearchResult]

    parse_document_jsonl

    source
    fn parse_document_jsonl(input : String) -> Result[Array[(String, String)], String]

    Parse JSONL documents with id or _id, optional title, and text. The complete batch is validated before the caller changes its index.

    parse_evaluation_cases

    source
    fn parse_evaluation_cases(text : String) -> Result[Array[EvaluationCase], String]

    parse_query

    source
    fn parse_query(query : String) -> Result[QueryPlan, String]

    Parse a compact query language:
    • +term: required term
    • -term: excluded term
    • "exact phrase": phrase clause
    • prefix*: prefix clause
    • title:moonbit: field-qualified clause

    precision_at

    source
    fn precision_at(ranked : Array[String], relevant : Array[String], cutoff : Int) -> Double

    r_precision

    source
    fn r_precision(ranked : Array[String], relevant : Array[String]) -> Double

    recall_at

    source
    fn recall_at(ranked : Array[String], relevant : Array[String], cutoff : Int) -> Double

    reciprocal_rank_at

    source
    fn reciprocal_rank_at(ranked : Array[String], relevant : Array[String], cutoff : Int) -> Double

    result_ids

    source
    fn result_ids(results : Array[SearchResult]) -> Array[String]

    result_score_range

    source
    fn result_score_range(results : Array[SearchResult]) -> (Double, Double)?

    result_snippet

    source
    fn result_snippet(text : String, query : String, max_chars? : Int, tokenizer? : Tokenizer) -> String

    Create a bounded result snippet centered near the first query match.

    rrf_fuse

    source
    fn rrf_fuse(a : Array[ScoredDoc], b : Array[ScoredDoc], k? : Int, top_k? : Int) -> Array[ScoredDoc]

    用 RRF（Reciprocal Rank Fusion）融合两路检索结果。

    shared_prefix_length

    source
    fn shared_prefix_length(left : String, right : String) -> Int

    snippets_for_results

    source
    fn snippets_for_results(results : Array[SearchResult], query : String, max_chars? : Int, tokenizer? : Tokenizer) -> Array[SearchResult]

    sort_results

    source
    fn sort_results(results : Array[SearchResult]) -> Array[SearchResult]

    source_doc_id

    source
    fn source_doc_id(chunk_id : String) -> String

    split_sentences

    source
    fn split_sentences(text : String) -> Array[String]

    Split prose into non-empty sentences while retaining terminal punctuation.

    suggest_terms

    source
    fn suggest_terms(query : String, vocabulary : Array[String], max_distance? : Int, limit? : Int) -> Array[QuerySuggestion]

    Suggest vocabulary entries within max_distance, ranked by distance, similarity, then lexical order for deterministic output.

    summarize_evaluations

    source
    fn summarize_evaluations(evaluations : Array[RetrievalMetrics]) -> EvaluationSummary

    summarize_text

    source
    fn summarize_text(text : String, query? : String, max_sentences? : Int, tokenizer? : Tokenizer) -> TextSummary

    Select the most query-relevant sentences, then restore document order.

    token_jaccard

    source
    fn token_jaccard(left : String, right : String, tokenizer? : Tokenizer) -> Double

    transposition_distance

    source
    fn transposition_distance(left : String, right : String) -> Int

    Optimal-string-alignment distance, treating one adjacent transposition as one edit. This is useful for common typing errors such as serach.

    weighted_fuse_results

    source
    fn weighted_fuse_results(left : Array[SearchResult], right : Array[SearchResult], left_weight? : Double, right_weight? : Double, limit? : Int) -> Array[SearchResult]

    Weighted score fusion. Duplicate IDs within one list keep their highest contribution; text comes from the highest-scored occurrence.

    Source Files

    Powered by MoonBit

    Site sourceReport issuePackagesBuild queueSkillsStatistics

    © 2026 mooncakes.io