Semantic chunking for RAG: embedding-similarity-based document splitting with a pluggable Embedder contract.
moon add yuzhiblue/moon-semchunk///|
fn embed(text : String) -> Array[Double] {
// Replace with a call to your embedding provider.
[1.0, 0.0]
}
///|
fn main {
let text = "The cat sleeps. The cat plays. The ocean is deep. Ships cross it."
let chunks = @moon_semchunk.chunk(
text,
embed,
@moon_semchunk.Percentile(95.0),
)
for chunk in chunks {
println(chunk.text)
}
}pub(all) struct Chunk {
text : String
index : Int
}pub(all) enum Threshold {
Percentile(Double)
StandardDeviation(Double)
Interquartile(Double)
}Install
Download zipSemantic chunking for RAG: embedding-similarity-based document splitting with a pluggable Embedder contract.