🥮 mooncakes.io

moonrag-bench

MoonRAGBench is a MoonBit-native toolkit for retrieval and RAG benchmark evaluation.

moonbit
rag
benchmark
retrieval
cli
moon add q0w1ertyuiop/moonrag-bench@0.2.0
Download zip
Version
0.2.0
License
Apache-2.0
Repository
Last updated
2 hours ago
Downloads
2

Dependencies

README

#MoonRAGBench

MoonRAGBench is a MoonBit-native toolkit for evaluating retrieval systems and RAG candidate generation offline. It turns qrels, ranked runs, and candidate pools into deterministic metrics, diagnostics, negative samples, and exportable reports.

It is useful for retrieval experiments, reranker iteration, small benchmark construction, course projects, and CI quality gates. The library is network-free: the same TSV inputs produce the same ordered output on every supported target.

#Core capabilities

  • Parse qrels, retrieval runs, and candidate pools from comment-tolerant TSV.
  • Validate identifiers, relevance values, finite scores, duplicates, and query coverage before evaluation.
  • Compute Recall, Precision, F1, R-precision, MRR, MAP, nDCG, BPref, ERR, RBP, fallout, judged coverage, and graded-gain diagnostics at multiple cutoffs.
  • Compare baseline and candidate runs with per-query wins/losses/ties, deltas, overlap, and rank movement.
  • Build dataset profiles, query/document statistics, relevance histograms, candidate recall, deterministic partitions, and negative-sample plans.
  • Export text, Markdown, CSV, JSON, JSON Lines, HTML, TSV traces, and reproducibility cards.
  • Use a native CLI while keeping the evaluation core pure and independently testable.

#Quick start

Install the module from Mooncakes:

moon add q0w1ertyuiop/moonrag-bench

Run the bundled demo from a checkout:

moon run cmd/main --target native -- demo

Evaluate a run:

moon run cmd/main --target native -- eval \--qrels examples/demo/qrels.tsv \--run examples/demo/run.tsv \--format markdown

#CLI

eval --qrels FILE --run FILE [--format text|markdown|json|csv|jsonl]
validate --qrels FILE --run FILE [--format text|markdown|json]
inspect --qrels FILE --run FILE
compare --qrels FILE --baseline FILE --candidate FILE [--cutoff N]
sample-negatives --qrels FILE --run FILE [--pool FILE]

Examples:

moon run cmd/main --target native -- validate \
  --qrels examples/benchmark/qrels.tsv \
  --run examples/benchmark/run_reranked.tsv

moon run cmd/main --target native -- compare \
  --qrels examples/benchmark/qrels.tsv \
  --baseline examples/benchmark/run_baseline.tsv \
  --candidate examples/benchmark/run_reranked.tsv \
  --cutoff 3

moon run cmd/main --target native -- sample-negatives \
  --qrels examples/benchmark/qrels.tsv \
  --run examples/benchmark/run_baseline.tsv \
  --pool examples/benchmark/pool.tsv \
  --count 2 --strategy hard --window 3

#Input formats

Qrels contain one judgment per line:

query_id<TAB>doc_id<TAB>relevance

Runs contain score-ranked candidates:

query_id<TAB>doc_id<TAB>score

Candidate pools contain query/document pairs:

query_id<TAB>doc_id

Blank lines and lines beginning with # are ignored. Scores must be finite for strict parsing. Ties are ordered by document identifier, so output does not depend on map iteration order.

#Architecture

qrels/run/pool TSV
       │
       ├── parse + strict validation + normalization
       │
       ├── metrics / comparison / dataset statistics / sampling
       │
       └── reports: text · Markdown · CSV · JSON · JSONL · HTML

The root package owns the public data model and pure evaluation APIs. The cmd/main package handles filesystem reads, argument parsing, and process output. Focused .mbt files separate parsing, metrics, comparison, sampling, analysis, reports, presets, and release helpers.

#Reproducible benchmark

The checked-in fixture under examples/benchmark contains eight queries with graded and non-relevant judgments, a deliberately weaker baseline, a reranked candidate, and candidate pools. It is project benchmark data for testing behavior and comparing revisions; it is not presented as an external corpus or production quality claim.

Regenerate the measurements:

./scripts/benchmark.ps1

The exact inputs and recorded output are documented in docs/benchmarks/results.md.

#Tests and CI

Run the local quality gate:

moon fmt --check
moon check --target all --deny-warn
moon test --target wasm --deny-warn
moon test --target wasm-gc --deny-warn
moon test --target js --deny-warn
moon info
moon run cmd/main --target native -- demo

GitHub Actions repeats the stable-toolchain checks on Ubuntu, macOS, and Windows, verifies generated interfaces and formatting, runs the portable library test targets, and exercises the native CLI. The workflow keeps native CLI smoke tests separate from portable library tests so process integration is checked explicitly.

#License

Apache-2.0. See LICENSE.

#Publishing

Update the version in moon.mod, run the local quality gate, and publish with the authenticated MoonBit CLI:

moon publish

The repository also contains a manual GitHub Actions publishing workflow. It expects a repository secret and never stores credentials in source control.

#
AggregateMetric

source
pub struct AggregateMetric {
  name : String
  mean : Double
  min : Double
  max : Double
} derive(ToJson, 
Debug)

#
BenchmarkCase

source
pub struct BenchmarkCase {
  name : String
  qrels : Array[JudgedDoc]
  run : Array[RetrievedDoc]
} derive(ToJson, 
Debug)

#
BenchmarkCase::new

source
fn BenchmarkCase::new(name~ : String, qrels~ : Array[JudgedDoc], run~ : Array[RetrievedDoc]) -> BenchmarkCase

#
BenchmarkManifest

source
pub struct BenchmarkManifest {
  name : String
  version : String
  license : String
  query_count : Int
  qrels_rows : Int
  run_rows : Int
  relevant_documents : Int
  candidate_coverage : Double
} derive(ToJson, 
Debug)

#
BenchmarkManifest::new

source
fn BenchmarkManifest::new(name~ : String, version~ : String, license~ : String, qrels~ : Array[JudgedDoc], run~ : Array[RetrievedDoc]) -> BenchmarkManifest

#
BenchmarkReport

source
pub struct BenchmarkReport {
  cutoffs : Array[Int]
  relevant_threshold : Int
  query_count : Int
  queries : Array[QueryEvaluation]
  summary : Array[AggregateMetric]
} derive(ToJson, 
Debug)

#
BenchmarkResult

source
pub struct BenchmarkResult {
  name : String
  report : BenchmarkReport
  validation : ValidationSummary
} derive(ToJson, 
Debug)

#
CandidatePool

source
pub struct CandidatePool {
  query_id : String
  doc_ids : Array[String]
} derive(ToJson, 
Debug)

#
CandidatePool::new

source
fn CandidatePool::new(query_id~ : String, doc_ids~ : Array[String]) -> CandidatePool

#
CandidatePoolProfile

source
pub struct CandidatePoolProfile {
  pool_count : Int
  total_candidate_count : Int
  unique_candidate_count : Int
  duplicate_candidate_count : Int
  mean_pool_size : Double
  min_pool_size : Int
  max_pool_size : Int
} derive(ToJson, 
Debug)

#
CliFormat

source
pub enum CliFormat {
  Text
  Markdown
  Json
  Csv
  JsonLines
} derive(Eq, ToJson, 
Debug)

#
CliRequest

source
pub enum CliRequest {
  Demo
  Eval(qrels_path~ : String, run_path~ : String, format~ : CliFormat, cutoffs~ : Array[Int], threshold~ : Int, gain~ : GainScheme)
  Validate(qrels_path~ : String, run_path~ : String, format~ : CliFormat)
  Compare(qrels_path~ : String, baseline_path~ : String, candidate_path~ : String, cutoff~ : Int)
  Inspect(qrels_path~ : String, run_path~ : String)
} derive(ToJson, 
Debug)

#
CorpusProfile

source
pub struct CorpusProfile {
  query_count : Int
  qrels_rows : Int
  run_rows : Int
  unique_document_count : Int
  relevant_document_count : Int
  max_relevance : Int
  unjudged_retrievals : Int
  mean_judgments_per_query : Double
  mean_run_length : Double
  mean_score : Double
  score_stddev : Double
} derive(ToJson, 
Debug)

#
DatasetProfile

source
pub struct DatasetProfile {
  query_count : Int
  judged_count : Int
  retrieved_count : Int
  relevant_count : Int
  run_query_coverage : Double
  qrels_query_coverage : Double
  mean_run_length : Double
  mean_score : Double
  score_stddev : Double
  unjudged_retrievals : Int
  empty_query_count : Int
  duplicate_query_count : Int
} derive(ToJson, 
Debug)

#
DatasetProfile::is_usable

source
fn DatasetProfile::is_usable(self : DatasetProfile) -> Bool

#
DocumentStatistic

source
pub struct DocumentStatistic {
  doc_id : String
  query_frequency : Int
  relevant_query_count : Int
  maximum_relevance : Int
} derive(ToJson, 
Debug)

#
EvalConfig

source
pub struct EvalConfig {
  cutoffs : Array[Int]
  relevant_threshold : Int
  gain_scheme : GainScheme
  missing_relevance : Int
} derive(ToJson, 
Debug)

#
EvalConfig::default

source
fn EvalConfig::default() -> EvalConfig

#
EvalConfig::new

source
fn EvalConfig::new(cutoffs~ : Array[Int], relevant_threshold? : Int, gain_scheme? : GainScheme, missing_relevance? : Int) -> EvalConfig

#
EvaluationPlan

source
pub struct EvaluationPlan {
  name : String
  cutoffs : Array[Int]
  threshold : Int
  gain : GainScheme
} derive(ToJson, 
Debug)

#
EvaluationPlan::new

source
fn EvaluationPlan::new(name~ : String, cutoffs~ : Array[Int], threshold~ : Int, gain~ : GainScheme) -> EvaluationPlan

#
GainScheme

source
pub enum GainScheme {
  Linear
  Exp2
} derive(Eq, ToJson, 
Debug)

#
GainScheme::exp2

source
fn GainScheme::exp2() -> GainScheme

#
GainScheme::linear

source
fn GainScheme::linear() -> GainScheme

#
JudgedDoc

source
pub struct JudgedDoc {
  query_id : String
  doc_id : String
  relevance : Int
} derive(Eq, ToJson, 
Debug)

#
JudgedDoc::new

source
fn JudgedDoc::new(query_id~ : String, doc_id~ : String, relevance~ : Int) -> JudgedDoc

#
MetricDistribution

source
pub struct MetricDistribution {
  count : Int
  mean : Double
  min : Double
  max : Double
  median : Double
  p25 : Double
  p75 : Double
  stddev : Double
} derive(ToJson, 
Debug)

#
NegativeSample

source
pub struct NegativeSample {
  query_id : String
  doc_id : String
  source_rank : Int
  strategy : String
} derive(Eq, ToJson, 
Debug)

#
NegativeSampleConfig

source
pub struct NegativeSampleConfig {
  per_query : Int
  relevant_threshold : Int
  skip_judged : Bool
  strategy : NegativeStrategy
} derive(ToJson, 
Debug)

#
NegativeSampleConfig::default

source

#
NegativeSampleConfig::new

source
fn NegativeSampleConfig::new(per_query~ : Int, relevant_threshold? : Int, skip_judged? : Bool, strategy? : NegativeStrategy) -> NegativeSampleConfig

#
NegativeStrategy

source
pub enum NegativeStrategy {
  Tail(Int)
  HardWindow(Int)
  Stride(Int)
} derive(Eq, ToJson, 
Debug)

#
NegativeStrategy::hard_window

source
fn NegativeStrategy::hard_window(window : Int) -> NegativeStrategy

#
NegativeStrategy::stride

source
fn NegativeStrategy::stride(step : Int) -> NegativeStrategy

#
NegativeStrategy::tail

source
fn NegativeStrategy::tail(window : Int) -> NegativeStrategy

#
PlanResult

source
pub struct PlanResult {
  name : String
  report : BenchmarkReport
  validation : ValidationSummary
} derive(ToJson, 
Debug)

#
QueryBucket

source
pub struct QueryBucket {
  query_id : String
  label : String
  score : Double
} derive(ToJson, 
Debug)

#
QueryComparison

source
pub struct QueryComparison {
  query_id : String
  baseline : Double
  candidate : Double
  delta : Double
  outcome : String
  overlap : Int
} derive(ToJson, 
Debug)

#
QueryEvaluation

source
pub struct QueryEvaluation {
  query_id : String
  relevant_total : Int
  retrieved_total : Int
  metrics : Map[String, Double]
} derive(ToJson, 
Debug)

#
QueryPartition

source
pub struct QueryPartition {
  partition : Int
  query_ids : Array[String]
  qrels : Array[JudgedDoc]
} derive(ToJson, 
Debug)

#
QueryStatistic

source
pub struct QueryStatistic {
  query_id : String
  judged_count : Int
  relevant_count : Int
  retrieved_count : Int
  unjudged_count : Int
  mean_score : Double
} derive(ToJson, 
Debug)

#
RankPoint

source
pub struct RankPoint {
  rank : Int
  precision : Double
  recall : Double
} derive(ToJson, 
Debug)

#
RetrievedDoc

source
pub struct RetrievedDoc {
  query_id : String
  doc_id : String
  score : Double
} derive(Eq, ToJson, 
Debug)

#
RetrievedDoc::new

source
fn RetrievedDoc::new(query_id~ : String, doc_id~ : String, score~ : Double) -> RetrievedDoc

#
RunComparison

source
pub struct RunComparison {
  cutoff : Int
  query_count : Int
  wins : Int
  losses : Int
  ties : Int
  mean_delta : Double
  queries : Array[QueryComparison]
} derive(ToJson, 
Debug)

#
RunOrder

source
pub enum RunOrder {
  ScoreDescending
  DocumentAscending
  InputOrder
} derive(Eq, ToJson, 
Debug)

#
RunOrder::document_ascending

source
fn RunOrder::document_ascending() -> RunOrder

#
RunOrder::input_order

source
fn RunOrder::input_order() -> RunOrder

#
RunOrder::score_descending

source
fn RunOrder::score_descending() -> RunOrder

#
RunQuality

source
pub struct RunQuality {
  query_count : Int
  row_count : Int
  unique_row_count : Int
  duplicate_rate : Double
  unjudged_rate : Double
  score_monotonicity : Double
  finite_score_rate : Double
} derive(ToJson, 
Debug)

#
ThresholdEvaluation

source
pub struct ThresholdEvaluation {
  threshold : Int
  cutoff : Int
  precision : Double
  recall : Double
  f1 : Double
  ndcg : Double
} derive(ToJson, 
Debug)

#
TraceStep

source
pub struct TraceStep {
  rank : Int
  doc_id : String
  score : Double
  relevance : Int
  judged : Bool
  hits : Int
  precision : Double
  recall : Double
} derive(ToJson, 
Debug)

#
TsvScanStats

source
pub struct TsvScanStats {
  total_lines : Int
  blank_lines : Int
  comment_lines : Int
  data_lines : Int
} derive(Eq, ToJson, 
Debug)

#
ValidationIssue

source
pub struct ValidationIssue {
  code : String
  level : ValidationLevel
  message : String
  query_id : String
  doc_id : String
} derive(Eq, ToJson, 
Debug)

#
ValidationLevel

source
pub enum ValidationLevel {
  Warning
  Error
} derive(Eq, ToJson, 
Debug)

#
ValidationSummary

source
pub struct ValidationSummary {
  error_count : Int
  warning_count : Int
  query_count : Int
  judged_count : Int
  retrieved_count : Int
  issues : Array[ValidationIssue]
} derive(ToJson, 
Debug)

#
ValidationSummary::is_valid

source
fn ValidationSummary::is_valid(self : ValidationSummary) -> Bool

#
API_VERSION

source
let API_VERSION : String

#
DATA_SCHEMA_VERSION

source
let DATA_SCHEMA_VERSION : String

#
api_healthcheck

source
fn api_healthcheck() -> String

#
available_presets

source
fn available_presets() -> Array[String]

#
average_precision_at

source
fn average_precision_at(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int) -> Double

#
benchmark_case_is_reproducible

source
fn benchmark_case_is_reproducible(case : BenchmarkCase, cutoffs~ : Array[Int]) -> Bool

#
benchmark_metric_table

source
fn benchmark_metric_table(result : BenchmarkResult) -> String

#
best_query

source
fn best_query(report : BenchmarkReport, metric : String) -> String

#
best_threshold

source
fn best_threshold(rows : Array[ThresholdEvaluation], cutoff : Int) -> Int

#
blend_runs

source
fn blend_runs(left : Array[RetrievedDoc], right : Array[RetrievedDoc], left_weight? : Double) -> Array[RetrievedDoc]

#
bottom_query_ids

source
fn bottom_query_ids(report : BenchmarkReport, metric : String, limit : Int) -> Array[String]

#
bpref_at

source
fn bpref_at(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int) -> Double

#
build_candidate_pools

source
fn build_candidate_pools(run : Array[RetrievedDoc]) -> Array[CandidatePool]

#
bundle_is_reproducible

source
fn bundle_is_reproducible(case : BenchmarkCase, cutoffs~ : Array[Int]) -> Bool

#
candidate_pool_coverage

source
fn candidate_pool_coverage(qrels : Array[JudgedDoc], pools : Array[CandidatePool]) -> Double

#
candidate_recall

source
fn candidate_recall(qrels : Array[JudgedDoc], pools : Array[CandidatePool]) -> Double

#
canonical_output_format

source
fn canonical_output_format(format : String) -> Result[CliFormat, String]

#
classify_queries

source
fn classify_queries(report : BenchmarkReport, cutoff~ : Int) -> Array[QueryBucket]

#
cli_format_name

source
fn cli_format_name(format : CliFormat) -> String

#
cli_request_summary

source
fn cli_request_summary(request : CliRequest) -> String

#
cli_usage_text

source
fn cli_usage_text() -> String

#
compare_benchmark_cases

source
fn compare_benchmark_cases(baseline : BenchmarkCase, candidate : BenchmarkCase, cutoff~ : Int) -> RunComparison

#
compare_metric_means

source
fn compare_metric_means(left : BenchmarkReport, right : BenchmarkReport, metric : String) -> Double

#
compare_runs

source
fn compare_runs(qrels : Array[JudgedDoc], baseline : Array[RetrievedDoc], candidate : Array[RetrievedDoc], cutoff~ : Int) -> RunComparison

#
config_signature

source
fn config_signature(config : EvalConfig) -> String

#
corpus_profile

source
fn corpus_profile(qrels : Array[JudgedDoc], run : Array[RetrievedDoc]) -> CorpusProfile

#
corpus_profile_json

source
fn corpus_profile_json(profile : CorpusProfile) -> String

#
coverage_at

source
fn coverage_at(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int) -> Double

#
deduplicate_candidate_pool

source
fn deduplicate_candidate_pool(pool : CandidatePool) -> CandidatePool

#
deduplicate_qrels

source
fn deduplicate_qrels(qrels : Array[JudgedDoc]) -> Array[JudgedDoc]

#
deduplicate_run

source
fn deduplicate_run(run : Array[RetrievedDoc]) -> Array[RetrievedDoc]

#
default_cutoffs

source
fn default_cutoffs() -> Array[Int]

#
demo_bundle

source
fn demo_bundle() -> String

#
demo_fixture

source
fn demo_fixture() -> BenchmarkCase

#
demo_fixture_pools

source
fn demo_fixture_pools() -> Array[CandidatePool]

#
demo_pool_source

source
fn demo_pool_source() -> String

#
demo_qrels_source

source
fn demo_qrels_source() -> String

#
demo_report

source
fn demo_report() -> BenchmarkReport

#
demo_run_source

source
fn demo_run_source() -> String

#
demo_validation

source
fn demo_validation() -> ValidationSummary

#
diagnose_run

source
fn diagnose_run(qrels : Array[JudgedDoc], run : Array[RetrievedDoc]) -> String

#
document_relevance_levels

source
fn document_relevance_levels(qrels : Array[JudgedDoc]) -> Array[Int]

#
document_statistics

source
fn document_statistics(qrels : Array[JudgedDoc]) -> Array[DocumentStatistic]

#
err_at

source
fn err_at(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int) -> Double

#
evaluate_benchmark

source
fn evaluate_benchmark(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], config? : EvalConfig) -> BenchmarkReport

#
evaluate_metric_family

source
fn evaluate_metric_family(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int, gain_scheme : GainScheme) -> Map[String, Double]

#
evaluate_query

source
fn evaluate_query(query_id : String, qrels : Array[JudgedDoc], run : Array[RetrievedDoc], config? : EvalConfig) -> QueryEvaluation

#
evaluate_with_preset

source
fn evaluate_with_preset(name : String, qrels : Array[JudgedDoc], run : Array[RetrievedDoc]) -> Result[BenchmarkReport, String]

#
example_eval_summary

source
fn example_eval_summary() -> String

#
example_negative_sample_summary

source
fn example_negative_sample_summary() -> String

#
example_profile_summary

source
fn example_profile_summary() -> String

#
example_quality_summary

source
fn example_quality_summary() -> String

#
execute_plan

source
fn execute_plan(plan : EvaluationPlan, qrels : Array[JudgedDoc], run : Array[RetrievedDoc]) -> PlanResult

#
execute_plan_with_profile

source
fn execute_plan_with_profile(plan : EvaluationPlan, qrels : Array[JudgedDoc], run : Array[RetrievedDoc]) -> String

#
export_bundle

source
fn export_bundle(case : BenchmarkCase, cutoffs~ : Array[Int]) -> String

#
export_bundle_json

source
fn export_bundle_json(case : BenchmarkCase, cutoffs~ : Array[Int]) -> String

#
export_report_files

source
fn export_report_files(report : BenchmarkReport) -> Map[String, String]

#
export_report_index

source
fn export_report_index(report : BenchmarkReport) -> String

#
f1_at

source
fn f1_at(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int) -> Double

#
f1_from_counts

source
fn f1_from_counts(relevant_retrieved : Int, retrieved : Int, relevant : Int) -> Double

#
fallout_at

source
fn fallout_at(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int) -> Double

#
filter_qrels_by_queries

source
fn filter_qrels_by_queries(qrels : Array[JudgedDoc], query_ids : Array[String]) -> Array[JudgedDoc]

#
filter_report_queries

source
fn filter_report_queries(report : BenchmarkReport, query_ids : Array[String]) -> BenchmarkReport

#
filter_run_by_queries

source
fn filter_run_by_queries(run : Array[RetrievedDoc], query_ids : Array[String]) -> Array[RetrievedDoc]

#
filter_run_by_score

source
fn filter_run_by_score(run : Array[RetrievedDoc], minimum_score : Double) -> Array[RetrievedDoc]

#
first_relevant_rank

source
fn first_relevant_rank(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], threshold : Int) -> Int

#
format_percent

source
fn format_percent(value : Double) -> String

#
graded_precision_at

source
fn graded_precision_at(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int, gain_scheme : GainScheme) -> Double

#
interleave_candidate_pools

source
fn interleave_candidate_pools(left : CandidatePool, right : CandidatePool) -> CandidatePool

#
is_known_preset

source
fn is_known_preset(name : String) -> Bool

#
is_supported_output_format

source
fn is_supported_output_format(format : String) -> Bool

#
jaccard_at

source
fn jaccard_at(left : Array[RetrievedDoc], right : Array[RetrievedDoc], query_id : String, cutoff : Int) -> Double

#
join_qrels_with_pool

source
fn join_qrels_with_pool(qrels : Array[JudgedDoc], pools : Array[CandidatePool]) -> Array[JudgedDoc]

#
judged_recall_at

source
fn judged_recall_at(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int) -> Double

#
manifest_with_pools

source
fn manifest_with_pools(name~ : String, version~ : String, license~ : String, qrels~ : Array[JudgedDoc], run~ : Array[RetrievedDoc], pools~ : Array[CandidatePool]) -> BenchmarkManifest

#
merge_candidate_pools

source
fn merge_candidate_pools(left : Array[CandidatePool], right : Array[CandidatePool]) -> Array[CandidatePool]

#
merge_runs

source
fn merge_runs(left : Array[RetrievedDoc], right : Array[RetrievedDoc]) -> Array[RetrievedDoc]

#
metric_aliases

source
fn metric_aliases() -> Map[String, String]

#
metric_mean_by_query

source
fn metric_mean_by_query(report : BenchmarkReport, prefix : String, cutoff : Int) -> Double

#
metric_rank

source
fn metric_rank(report : BenchmarkReport, metric : String, query_id : String) -> Int

#
metric_rank_delta

source
fn metric_rank_delta(baseline : BenchmarkReport, candidate : BenchmarkReport, metric : String, query_id : String) -> Int

#
metric_wins

source
fn metric_wins(baseline : BenchmarkReport, candidate : BenchmarkReport, name : String) -> (Int, Int, Int)

#
module_identifier

source
fn module_identifier() -> String

#
ndcg_at

source
fn ndcg_at(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int, gain_scheme : GainScheme) -> Double

#
normalize_cutoffs

source
fn normalize_cutoffs(cutoffs : Array[Int]) -> Array[Int]

#
normalize_inputs

source
fn normalize_inputs(qrels : Array[JudgedDoc], run : Array[RetrievedDoc]) -> (Array[JudgedDoc], Array[RetrievedDoc])

#
normalize_plan

source
fn normalize_plan(plan : EvaluationPlan) -> EvaluationPlan

#
normalize_run

source
fn normalize_run(run : Array[RetrievedDoc]) -> Array[RetrievedDoc]

#
normalize_scores

source
fn normalize_scores(run : Array[RetrievedDoc]) -> Array[RetrievedDoc]

#
overlap_at

source
fn overlap_at(baseline : Array[RetrievedDoc], candidate : Array[RetrievedDoc], query_id : String, cutoff : Int) -> Int

#
parse_candidate_pool_tsv

source
fn parse_candidate_pool_tsv(source : String) -> Result[Array[CandidatePool], String]

#
parse_cli_request

source
fn parse_cli_request(args : Array[String]) -> Result[CliRequest, String]

#
parse_qrels_tsv

source
fn parse_qrels_tsv(source : String) -> Result[Array[JudgedDoc], String]

#
parse_qrels_tsv_strict

source
fn parse_qrels_tsv_strict(source : String) -> Result[Array[JudgedDoc], String]

#
parse_run_tsv

source
fn parse_run_tsv(source : String) -> Result[Array[RetrievedDoc], String]

#
parse_run_tsv_strict

source
fn parse_run_tsv_strict(source : String) -> Result[Array[RetrievedDoc], String]

#
partition_qrels

source
fn partition_qrels(qrels : Array[JudgedDoc], partition_count : Int) -> Array[QueryPartition]

#
partition_row_count

source
fn partition_row_count(partitions : Array[QueryPartition]) -> Int

#
plan_has_metric

source
fn plan_has_metric(result : PlanResult, name : String) -> Bool

#
plan_metric

source
fn plan_metric(result : PlanResult, name : String) -> MetricDistribution

#
plan_sampling

source
fn plan_sampling(qrels : Array[JudgedDoc], pools : Array[CandidatePool], total_budget~ : Int) -> String

#
plan_to_json

source
fn plan_to_json(result : PlanResult) -> String

#
plan_validation_markdown

source
fn plan_validation_markdown(result : PlanResult) -> String

#
precision_at

source
fn precision_at(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int) -> Double

#
precision_curve

source
fn precision_curve(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int) -> Array[Double]

#
precision_from_counts

source
fn precision_from_counts(relevant_retrieved : Int, retrieved : Int) -> Double

#
precision_recall_area

source
fn precision_recall_area(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int) -> Double

#
precision_recall_points

source
fn precision_recall_points(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int) -> Array[RankPoint]

#
preset_config

source
fn preset_config(name : String) -> Result[EvalConfig, String]

#
preset_cutoffs

source
fn preset_cutoffs(name : String) -> Result[Array[Int], String]

#
preset_description

source
fn preset_description(name : String) -> Result[String, String]

#
profile_candidate_pools

source
fn profile_candidate_pools(pools : Array[CandidatePool]) -> CandidatePoolProfile

#
profile_dataset

source
fn profile_dataset(qrels : Array[JudgedDoc], run : Array[RetrievedDoc]) -> DatasetProfile

#
quality_gate

source
fn quality_gate(case : BenchmarkCase, report : BenchmarkReport) -> String

#
quantile

source
fn quantile(values : Array[Double], fraction : Double) -> Double

#
query_metric_rows

source
fn query_metric_rows(report : BenchmarkReport, metric_names : Array[String]) -> String

#
query_statistics

source
fn query_statistics(qrels : Array[JudgedDoc], run : Array[RetrievedDoc]) -> Array[QueryStatistic]

#
query_statistics_markdown

source
fn query_statistics_markdown(stats : Array[QueryStatistic]) -> String

#
r_precision

source
fn r_precision(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], threshold : Int) -> Double

#
rank_biased_overlap

source
fn rank_biased_overlap(left : Array[RetrievedDoc], right : Array[RetrievedDoc], query_id : String, cutoff : Int, persistence? : Double) -> Double

#
rank_movement

source
fn rank_movement(baseline : Array[RetrievedDoc], candidate : Array[RetrievedDoc], query_id : String, cutoff : Int) -> Int

#
rbp_at

source
fn rbp_at(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int, persistence? : Double) -> Double

#
recall_at

source
fn recall_at(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int) -> Double

#
recall_curve

source
fn recall_curve(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int) -> Array[Double]

#
recall_from_counts

source
fn recall_from_counts(relevant_retrieved : Int, relevant : Int) -> Double

#
reciprocal_rank_at

source
fn reciprocal_rank_at(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int) -> Double

#
release_channel

source
fn release_channel() -> String

#
release_checklist

source
fn release_checklist() -> Array[String]

#
release_metadata

source
fn release_metadata() -> String

#
release_notes

source
fn release_notes() -> String

#
release_target_count

source
fn release_target_count() -> Int

#
release_target_names

source
fn release_target_names() -> Array[String]

#
relevance_histogram

source
fn relevance_histogram(qrels : Array[JudgedDoc]) -> Map[Int, Int]

#
relevant_rank_positions

source
fn relevant_rank_positions(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], threshold : Int) -> Array[Int]

#
render_benchmark_manifest

source
fn render_benchmark_manifest(case : BenchmarkCase) -> String

#
render_benchmark_result

source
fn render_benchmark_result(result : BenchmarkResult) -> String

#
render_compact_report

source
fn render_compact_report(report : BenchmarkReport) -> String

#
render_comparison_json

source
fn render_comparison_json(comparison : RunComparison) -> String

#
render_comparison_markdown

source
fn render_comparison_markdown(comparison : RunComparison) -> String

#
render_comparison_text

source
fn render_comparison_text(comparison : RunComparison) -> String

#
render_corpus_profile

source
fn render_corpus_profile(profile : CorpusProfile) -> String

#
render_csv_report

source
fn render_csv_report(report : BenchmarkReport) -> String

#
render_distribution_markdown

source
fn render_distribution_markdown(name : String, distribution : MetricDistribution) -> String

#
render_histogram

source
fn render_histogram(histogram : Map[Int, Int]) -> String

#
render_html_report

source
fn render_html_report(report : BenchmarkReport) -> String

#
render_json_lines_report

source
fn render_json_lines_report(report : BenchmarkReport) -> String

#
render_json_report

source
fn render_json_report(report : BenchmarkReport) -> String

#
render_manifest

source
fn render_manifest(manifest : BenchmarkManifest) -> String

#
render_manifest_json

source
fn render_manifest_json(manifest : BenchmarkManifest) -> String

#
render_markdown_report

source
fn render_markdown_report(report : BenchmarkReport) -> String

#
render_metric_matrix

source
fn render_metric_matrix(reports : Array[(String, BenchmarkReport)]) -> String

#
render_negative_samples_tsv

source
fn render_negative_samples_tsv(samples : Array[NegativeSample]) -> String

#
render_plan_summary

source
fn render_plan_summary(result : PlanResult) -> String

#
render_pools_tsv

source
fn render_pools_tsv(pools : Array[CandidatePool]) -> String

#
render_preset_catalog

source
fn render_preset_catalog() -> String

#
render_preset_report

source
fn render_preset_report(name : String, report : BenchmarkReport) -> String

#
render_profile_json

source
fn render_profile_json(profile : DatasetProfile) -> String

#
render_profile_text

source
fn render_profile_text(profile : DatasetProfile) -> String

#
render_qrels_tsv

source
fn render_qrels_tsv(qrels : Array[JudgedDoc]) -> String

#
render_quality_summary

source
fn render_quality_summary(quality : RunQuality) -> String

#
render_query_buckets_markdown

source
fn render_query_buckets_markdown(buckets : Array[QueryBucket]) -> String

#
render_ranked_metric_table

source
fn render_ranked_metric_table(report : BenchmarkReport, metric : String) -> String

#
render_reproducibility_card

source
fn render_reproducibility_card(case : BenchmarkCase) -> String

#
render_run_tsv

source
fn render_run_tsv(run : Array[RetrievedDoc]) -> String

#
render_statistics_markdown

source
fn render_statistics_markdown(qrels : Array[JudgedDoc], run : Array[RetrievedDoc]) -> String

#
render_threshold_markdown

source
fn render_threshold_markdown(rows : Array[ThresholdEvaluation]) -> String

#
render_threshold_sweep

source
fn render_threshold_sweep(rows : Array[ThresholdEvaluation]) -> String

#
render_trace_tsv

source
fn render_trace_tsv(steps : Array[TraceStep]) -> String

#
render_validation_and_profile

source
fn render_validation_and_profile(qrels : Array[JudgedDoc], run : Array[RetrievedDoc]) -> String

#
render_validation_report

source
fn render_validation_report(summary : ValidationSummary) -> String

#
report_digest

source
fn report_digest(report : BenchmarkReport) -> String

#
report_has_finite_metrics

source
fn report_has_finite_metrics(report : BenchmarkReport) -> Bool

#
report_metric_names

source
fn report_metric_names(report : BenchmarkReport) -> Array[String]

#
resolve_metric_name

source
fn resolve_metric_name(prefix : String, cutoff : Int) -> String

#
retrieved_document_count

source
fn retrieved_document_count(run : Array[RetrievedDoc]) -> Int

#
run_benchmark_case

source
fn run_benchmark_case(case : BenchmarkCase, cutoffs~ : Array[Int]) -> BenchmarkResult

#
run_quality

source
fn run_quality(qrels : Array[JudgedDoc], run : Array[RetrievedDoc]) -> RunQuality

#
run_signature

source
fn run_signature(run : Array[RetrievedDoc]) -> String

#
run_to_pool

source
fn run_to_pool(run : Array[RetrievedDoc], query_id : String) -> CandidatePool

#
safe_compare

source
fn safe_compare(qrels : Array[JudgedDoc], baseline : Array[RetrievedDoc], candidate : Array[RetrievedDoc]) -> Result[RunComparison, String]

#
safe_evaluate

source
fn safe_evaluate(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], config? : EvalConfig) -> Result[BenchmarkReport, String]

#
safe_parse_and_evaluate

source
fn safe_parse_and_evaluate(qrels_source : String, run_source : String, config : EvalConfig) -> Result[BenchmarkReport, String]

#
safe_report_json

source
fn safe_report_json(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], config? : EvalConfig) -> Result[String, String]

#
safe_sample

source
fn safe_sample(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], config : NegativeSampleConfig) -> Result[Array[NegativeSample], String]

#
sample_hard_and_tail

source
fn sample_hard_and_tail(qrels : Array[JudgedDoc], pools : Array[CandidatePool], per_query~ : Int, window~ : Int) -> Array[NegativeSample]

#
sample_negatives

source
fn sample_negatives(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], pools? : Array[CandidatePool], config? : NegativeSampleConfig) -> Array[NegativeSample]

#
sample_pool_negatives

source
fn sample_pool_negatives(qrels : Array[JudgedDoc], pools : Array[CandidatePool], config? : NegativeSampleConfig) -> Array[NegativeSample]

#
sample_pool_negatives_tsv

source
fn sample_pool_negatives_tsv(qrels : Array[JudgedDoc], pools : Array[CandidatePool], config? : NegativeSampleConfig) -> String

#
sample_with_budget

source
fn sample_with_budget(qrels : Array[JudgedDoc], pools : Array[CandidatePool], total_budget~ : Int) -> Array[NegativeSample]

#
sampling_plan

source
fn sampling_plan(qrels : Array[JudgedDoc], pools : Array[CandidatePool], config? : NegativeSampleConfig) -> String

#
sampling_strategy_description

source
fn sampling_strategy_description(strategy : NegativeStrategy) -> String

#
sampling_yield

source
fn sampling_yield(qrels : Array[JudgedDoc], pools : Array[CandidatePool], samples : Array[NegativeSample]) -> Double

#
scan_tsv_source

source
fn scan_tsv_source(source : String) -> TsvScanStats

#
score_bucket_counts

source
fn score_bucket_counts(run : Array[RetrievedDoc], bucket_count : Int) -> Array[Int]

#
score_mean

source
fn score_mean(run : Array[RetrievedDoc]) -> Double

#
score_monotonicity

source
fn score_monotonicity(run : Array[RetrievedDoc]) -> Double

#
score_order_correlation

source
fn score_order_correlation(left : Array[RetrievedDoc], right : Array[RetrievedDoc], query_id : String, cutoff : Int) -> Double

#
score_stddev

source
fn score_stddev(run : Array[RetrievedDoc]) -> Double

#
select_queries_by_score

source
fn select_queries_by_score(report : BenchmarkReport, metric : String, minimum : Double) -> Array[String]

#
sort_run

source
fn sort_run(run : Array[RetrievedDoc], order : RunOrder) -> Array[RetrievedDoc]

#
summarize_report_metric

source
fn summarize_report_metric(report : BenchmarkReport, name : String) -> MetricDistribution

#
summarize_values

source
fn summarize_values(values : Array[Double]) -> MetricDistribution

#
supported_output_formats

source
fn supported_output_formats() -> Array[String]

#
threshold_frontier

source
fn threshold_frontier(rows : Array[ThresholdEvaluation]) -> Array[ThresholdEvaluation]

#
threshold_sweep

source
fn threshold_sweep(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], thresholds~ : Array[Int], cutoffs~ : Array[Int]) -> Array[ThresholdEvaluation]

#
top_query_ids

source
fn top_query_ids(report : BenchmarkReport, metric : String, limit : Int) -> Array[String]

#
trace_auc

source
fn trace_auc(steps : Array[TraceStep]) -> Double

#
trace_first_judged_rank

source
fn trace_first_judged_rank(steps : Array[TraceStep]) -> Int

#
trace_first_relevant_rank

source
fn trace_first_relevant_rank(steps : Array[TraceStep]) -> Int

#
trace_query

source
fn trace_query(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff~ : Int, threshold~ : Int) -> Array[TraceStep]

#
trimmed_mean

source
fn trimmed_mean(values : Array[Double], trim_fraction : Double) -> Double

#
truncate_run

source
fn truncate_run(run : Array[RetrievedDoc], per_query : Int) -> Array[RetrievedDoc]

#
unique_documents_at

source
fn unique_documents_at(run : Array[RetrievedDoc], query_id : String, cutoff : Int) -> Array[String]

#
validate_dataset

source
fn validate_dataset(qrels : Array[JudgedDoc], run : Array[RetrievedDoc]) -> ValidationSummary

#
validate_eval_config

source
fn validate_eval_config(config : EvalConfig) -> Result[Unit, String]

#
validate_manifest

source
fn validate_manifest(manifest : BenchmarkManifest) -> ValidationSummary

#
weighted_mean

source
fn weighted_mean(values : Array[Double], weights : Array[Double]) -> Double

#
worst_query

source
fn worst_query(report : BenchmarkReport, metric : String) -> String

Source Files

Powered by MoonBit

Site sourceReport issuePackagesBuild queueSkillsStatistics

© 2026 mooncakes.io