moon-semchunk

    Semantic chunking for RAG: embedding-similarity-based document splitting with a pluggable Embedder contract.

    rag
    semantic-chunking
    nlp
    text
    Download zip
    Author
    Version
    0.1.1
    License
    MIT
    Last updated
    13 hours ago
    Downloads
    8

    No dependencies

    #moon-semchunk

    Semantic chunking for RAG in pure MoonBit.

    Splits documents at semantic boundaries instead of fixed character counts. Sentences are embedded via a caller-supplied callback, adjacent cosine differences are computed, and statistical breakpoints decide where one chunk ends and the next begins.

    #Why

    Naive fixed-length or recursive splitting cuts mid-thought and harms retrieval quality. Semantic chunking keeps topically related sentences together. Existing MoonBit RAG libraries provide recursive / token / markdown / HTML splitters, but none implement similarity-based semantic boundaries. moon-semchunk fills that gap as a standalone, reusable library.

    Design follows LangChain's SemanticChunker (MIT, https://github.com/langchain-ai/langchain-text-splitters) and LlamaIndex's SemanticSplitterNodeParser (MIT, https://github.com/run-llama/llama_index), reimplemented natively in MoonBit. It is a port of the algorithm, not a copy of the source. Ported: statistical threshold computation, similarity-to-difference conversion and adjacent-sentence breakpoint detection. Not ported: buffer windows, token budgets, hierarchical node structures and LLM-based rewrites.

    #Install

    moon add yuzhiblue/moon-semchunk

    #Usage

    ///|
    fn embed(text : String) -> Array[Double] {
    // Replace with a call to your embedding provider.
    [1.0, 0.0]
    }

    ///|
    fn main {
    let text = "The cat sleeps. The cat plays. The ocean is deep. Ships cross it."
    let chunks = @moon_semchunk.chunk(
    text,
    embed,
    @moon_semchunk.Percentile(95.0),
    )
    for chunk in chunks {
    println(chunk.text)
    }
    }

    The library never calls a model itself. embedder : (String) -> Array[Double] is a pure callback, so the whole pipeline is deterministic and testable on every backend (native / wasm / wasm-gc / js).

    #API

    • chunk(text, embedder, strategy) — one-shot semantic splitting.
    • split_sentences(text) — sentence segmentation (.!?。!?).
    • cosine(a, b) — cosine similarity.
    • adjacent_differences(embeddings) — 1 - cosine between neighbours.
    • detect_breakpoints(differences, strategy) — boolean breakpoint mask.
    • merge_sentences(sentences, breakpoints) — assemble Chunk values.

    Threshold strategies:

    • Percentile(p) — break above the p-th percentile of differences (default 95).
    • StandardDeviation(k) — break above mean + k * stddev (default 3).
    • Interquartile(k) — break above Q3 + k * IQR (default 1.5).

    #License

    MIT. Algorithm references: LangChain SemanticChunker (MIT, https://github.com/langchain-ai/langchain-text-splitters) and LlamaIndex SemanticSplitterNodeParser (MIT, https://github.com/run-llama/llama_index). Source code was not copied.

    Chunk

    pub(all) struct Chunk {
    text : String
    index : Int
    }

    A chunk produced by semantic splitting.

    Threshold

    pub(all) enum Threshold {
    Percentile(Double)
    StandardDeviation(Double)
    Interquartile(Double)
    }

    Strategy for deciding where a semantic boundary lies.

    adjacent_differences

    fn adjacent_differences(embeddings : Array[Array[Double]]) -> Array[Double]

    Adjacent sentence differences from a list of embeddings.

    chunk

    fn chunk(text : String, embedder : (String) -> Array[Double], strategy : Threshold) -> Array[Chunk]

    Split text at semantic boundaries using a caller-supplied embedder.

    This is the one-shot entry point: it splits into sentences, obtains a vector for each sentence via embedder, computes adjacent differences, detects breakpoints with strategy, and merges sentences into chunks.

    The library never calls a model itself; embedder is a pure callback so the whole pipeline stays deterministic and testable on every backend.

    cosine

    fn cosine(a : Array[Double], b : Array[Double]) -> Double

    Cosine similarity between two vectors. Returns 0.0 when either vector has zero magnitude or lengths differ.

    detect_breakpoints

    fn detect_breakpoints(differences : Array[Double], strategy : Threshold) -> Array[Bool]

    Decide which adjacent positions are semantic breakpoints. Returns a boolean array of length differences.length().

    merge_sentences

    fn merge_sentences(sentences : Array[String], breakpoints : Array[Bool]) -> Array[Chunk]

    Merge sentences into chunks at breakpoint positions.

    breakpoints[i] is true when a boundary sits between sentence i and sentence i + 1. Sentences are joined with a single space.

    split_sentences

    fn split_sentences(text : String) -> Array[String]

    Break text into sentences at sentence-ending punctuation.

    Supported terminators: ., !, ?, 。, !, ?. The terminator is kept at the end of the sentence. Empty sentences are dropped.

    Source Files