🥮 mooncakes.io

    moonrag-bench

    MoonRAGBench is a MoonBit-native toolkit for retrieval and RAG benchmark evaluation.

    moonbit
    rag
    benchmark
    retrieval
    cli
    Download zip
    Version
    0.2.1
    License
    Apache-2.0
    Repository
    Last updated
    2 months ago
    Downloads
    34

    #MoonRAGBench

    MoonRAGBench is a MoonBit-native toolkit for evaluating retrieval systems and RAG candidate generation offline. It turns qrels, ranked runs, and candidate pools into deterministic metrics, diagnostics, negative samples, and exportable reports.

    It is useful for retrieval experiments, reranker iteration, small benchmark construction, course projects, and CI quality gates. The library is network-free: the same TSV inputs produce the same ordered output on every supported target.

    #Core capabilities

    • Parse qrels, retrieval runs, and candidate pools from comment-tolerant TSV.
    • Validate identifiers, relevance values, finite scores, duplicates, and query coverage before evaluation.
    • Compute Recall, Precision, F1, R-precision, MRR, MAP, nDCG, BPref, ERR, RBP, fallout, judged coverage, and graded-gain diagnostics at multiple cutoffs.
    • Compare baseline and candidate runs with per-query wins/losses/ties, deltas, overlap, and rank movement.
    • Build dataset profiles, query/document statistics, relevance histograms, candidate recall, deterministic partitions, and negative-sample plans.
    • Export text, Markdown, CSV, JSON, JSON Lines, HTML, TSV traces, and reproducibility cards.
    • Use a native CLI while keeping the evaluation core pure and independently testable.

    #Quick start

    Install the module from Mooncakes:

    moon add q0w1ertyuiop/moonrag-bench

    Run the bundled demo from a checkout:

    moon run cmd/main --target native -- demo

    Evaluate a run:

    moon run cmd/main --target native -- eval \--qrels examples/demo/qrels.tsv \--run examples/demo/run.tsv \--format markdown

    #CLI

    eval --qrels FILE --run FILE [--format text|markdown|json|csv|jsonl]
validate --qrels FILE --run FILE [--format text|markdown|json]
inspect --qrels FILE --run FILE
compare --qrels FILE --baseline FILE --candidate FILE [--cutoff N]
sample-negatives --qrels FILE --run FILE [--pool FILE]

    Examples:

    moon run cmd/main --target native -- validate \
  --qrels examples/benchmark/qrels.tsv \
  --run examples/benchmark/run_reranked.tsv

moon run cmd/main --target native -- compare \
  --qrels examples/benchmark/qrels.tsv \
  --baseline examples/benchmark/run_baseline.tsv \
  --candidate examples/benchmark/run_reranked.tsv \
  --cutoff 3

moon run cmd/main --target native -- sample-negatives \
  --qrels examples/benchmark/qrels.tsv \
  --run examples/benchmark/run_baseline.tsv \
  --pool examples/benchmark/pool.tsv \
  --count 2 --strategy hard --window 3

    #Input formats

    Qrels contain one judgment per line:

    query_id<TAB>doc_id<TAB>relevance

    Runs contain score-ranked candidates:

    query_id<TAB>doc_id<TAB>score

    Candidate pools contain query/document pairs:

    query_id<TAB>doc_id

    Blank lines and lines beginning with # are ignored. Scores must be finite for strict parsing. Ties are ordered by document identifier, so output does not depend on map iteration order.

    #Architecture

    qrels/run/pool TSV
       │
       ├── parse + strict validation + normalization
       │
       ├── metrics / comparison / dataset statistics / sampling
       │
       └── reports: text · Markdown · CSV · JSON · JSONL · HTML

    The root package owns the public data model and pure evaluation APIs. The cmd/main package handles filesystem reads, argument parsing, and process output. Focused .mbt files separate parsing, metrics, comparison, sampling, analysis, reports, presets, and release helpers.

    #Reproducible benchmark

    The checked-in fixture under examples/benchmark contains eight queries with graded and non-relevant judgments, a deliberately weaker baseline, a reranked candidate, and candidate pools. It is project benchmark data for testing behavior and comparing revisions; it is not presented as an external corpus or production quality claim.

    Regenerate the measurements:

    ./scripts/benchmark.ps1

    The exact inputs and recorded output are documented in docs/benchmarks/results.md.

    #Tests and CI

    Run the local quality gate:

    moon fmt --check
moon check --target all --deny-warn
moon test --target wasm --deny-warn
moon test --target wasm-gc --deny-warn
moon test --target js --deny-warn
moon info
moon run cmd/main --target native -- demo

    GitHub Actions repeats the stable-toolchain checks on Ubuntu, macOS, and Windows, verifies generated interfaces and formatting, runs the portable library test targets, and exercises the native CLI. The workflow keeps native CLI smoke tests separate from portable library tests so process integration is checked explicitly.

    #License

    Apache-2.0. See LICENSE.

    #Publishing

    Update the version in moon.mod, run the local quality gate, and publish with the authenticated MoonBit CLI:

    moon publish

    The repository also contains a manual GitHub Actions publishing workflow. It expects a repository secret and never stores credentials in source control.

    AggregateMetric

    source
    pub struct AggregateMetric {
      name : String
      mean : Double
      min : Double
      max : Double
    } derive(ToJson, 
    Debug    )

    BenchmarkCase

    source
    pub struct BenchmarkCase {
      name : String
      qrels : Array[JudgedDoc]
      run : Array[RetrievedDoc]
    } derive(ToJson, 
    Debug    )

    BenchmarkCase::new

    source
    fn BenchmarkCase::new(name~ : String, qrels~ : Array[JudgedDoc], run~ : Array[RetrievedDoc]) -> BenchmarkCase

    BenchmarkManifest

    source
    pub struct BenchmarkManifest {
      name : String
      version : String
      license : String
      query_count : Int
      qrels_rows : Int
      run_rows : Int
      relevant_documents : Int
      candidate_coverage : Double
    } derive(ToJson, 
    Debug    )

    BenchmarkManifest::new

    source
    fn BenchmarkManifest::new(name~ : String, version~ : String, license~ : String, qrels~ : Array[JudgedDoc], run~ : Array[RetrievedDoc]) -> BenchmarkManifest

    BenchmarkReport

    source
    pub struct BenchmarkReport {
      cutoffs : Array[Int]
      relevant_threshold : Int
      query_count : Int
      queries : Array[QueryEvaluation]
      summary : Array[AggregateMetric]
    } derive(ToJson, 
    Debug    )

    BenchmarkResult

    source
    pub struct BenchmarkResult {
      name : String
      report : BenchmarkReport
      validation : ValidationSummary
    } derive(ToJson, 
    Debug    )

    CandidatePool

    source
    pub struct CandidatePool {
      query_id : String
      doc_ids : Array[String]
    } derive(ToJson, 
    Debug    )

    CandidatePool::new

    source
    fn CandidatePool::new(query_id~ : String, doc_ids~ : Array[String]) -> CandidatePool

    CandidatePoolProfile

    source
    pub struct CandidatePoolProfile {
      pool_count : Int
      total_candidate_count : Int
      unique_candidate_count : Int
      duplicate_candidate_count : Int
      mean_pool_size : Double
      min_pool_size : Int
      max_pool_size : Int
    } derive(ToJson, 
    Debug    )

    CliFormat

    source
    pub enum CliFormat {
      Text
      Markdown
      Json
      Csv
      JsonLines
    } derive(Eq, ToJson, 
    Debug    )

    CliRequest

    source
    pub enum CliRequest {
      Demo
      Eval(qrels_path~ : String, run_path~ : String, format~ : CliFormat, cutoffs~ : Array[Int], threshold~ : Int, gain~ : GainScheme)
      Validate(qrels_path~ : String, run_path~ : String, format~ : CliFormat)
      Compare(qrels_path~ : String, baseline_path~ : String, candidate_path~ : String, cutoff~ : Int)
      Inspect(qrels_path~ : String, run_path~ : String)
    } derive(ToJson, 
    Debug    )

    CorpusProfile

    source
    pub struct CorpusProfile {
      query_count : Int
      qrels_rows : Int
      run_rows : Int
      unique_document_count : Int
      relevant_document_count : Int
      max_relevance : Int
      unjudged_retrievals : Int
      mean_judgments_per_query : Double
      mean_run_length : Double
      mean_score : Double
      score_stddev : Double
    } derive(ToJson, 
    Debug    )

    DatasetProfile

    source
    pub struct DatasetProfile {
      query_count : Int
      judged_count : Int
      retrieved_count : Int
      relevant_count : Int
      run_query_coverage : Double
      qrels_query_coverage : Double
      mean_run_length : Double
      mean_score : Double
      score_stddev : Double
      unjudged_retrievals : Int
      empty_query_count : Int
      duplicate_query_count : Int
    } derive(ToJson, 
    Debug    )

    DatasetProfile::is_usable

    source
    fn DatasetProfile::is_usable(self : DatasetProfile) -> Bool

    DocumentStatistic

    source
    pub struct DocumentStatistic {
      doc_id : String
      query_frequency : Int
      relevant_query_count : Int
      maximum_relevance : Int
    } derive(ToJson, 
    Debug    )

    EvalConfig

    source
    pub struct EvalConfig {
      cutoffs : Array[Int]
      relevant_threshold : Int
      gain_scheme : GainScheme
      missing_relevance : Int
    } derive(ToJson, 
    Debug    )

    EvalConfig::default

    source
    fn EvalConfig::default() -> EvalConfig

    EvalConfig::new

    source
    fn EvalConfig::new(cutoffs~ : Array[Int], relevant_threshold? : Int, gain_scheme? : GainScheme, missing_relevance? : Int) -> EvalConfig

    EvaluationPlan

    source
    pub struct EvaluationPlan {
      name : String
      cutoffs : Array[Int]
      threshold : Int
      gain : GainScheme
    } derive(ToJson, 
    Debug    )

    EvaluationPlan::new

    source
    fn EvaluationPlan::new(name~ : String, cutoffs~ : Array[Int], threshold~ : Int, gain~ : GainScheme) -> EvaluationPlan

    GainScheme

    source
    pub enum GainScheme {
      Linear
      Exp2
    } derive(Eq, ToJson, 
    Debug    )

    GainScheme::exp2

    source
    fn GainScheme::exp2() -> GainScheme

    GainScheme::linear

    source
    fn GainScheme::linear() -> GainScheme

    JudgedDoc

    source
    pub struct JudgedDoc {
      query_id : String
      doc_id : String
      relevance : Int
    } derive(Eq, ToJson, 
    Debug    )

    JudgedDoc::new

    source
    fn JudgedDoc::new(query_id~ : String, doc_id~ : String, relevance~ : Int) -> JudgedDoc

    MetricDistribution

    source
    pub struct MetricDistribution {
      count : Int
      mean : Double
      min : Double
      max : Double
      median : Double
      p25 : Double
      p75 : Double
      stddev : Double
    } derive(ToJson, 
    Debug    )

    NegativeSample

    source
    pub struct NegativeSample {
      query_id : String
      doc_id : String
      source_rank : Int
      strategy : String
    } derive(Eq, ToJson, 
    Debug    )

    NegativeSampleConfig

    source
    pub struct NegativeSampleConfig {
      per_query : Int
      relevant_threshold : Int
      skip_judged : Bool
      strategy : NegativeStrategy
    } derive(ToJson, 
    Debug    )

    NegativeSampleConfig::default

    source

    NegativeSampleConfig::new

    source
    fn NegativeSampleConfig::new(per_query~ : Int, relevant_threshold? : Int, skip_judged? : Bool, strategy? : NegativeStrategy) -> NegativeSampleConfig

    NegativeStrategy

    source
    pub enum NegativeStrategy {
      Tail(Int)
      HardWindow(Int)
      Stride(Int)
    } derive(Eq, ToJson, 
    Debug    )

    NegativeStrategy::hard_window

    source
    fn NegativeStrategy::hard_window(window : Int) -> NegativeStrategy

    NegativeStrategy::stride

    source
    fn NegativeStrategy::stride(step : Int) -> NegativeStrategy

    NegativeStrategy::tail

    source
    fn NegativeStrategy::tail(window : Int) -> NegativeStrategy

    PlanResult

    source
    pub struct PlanResult {
      name : String
      report : BenchmarkReport
      validation : ValidationSummary
    } derive(ToJson, 
    Debug    )

    QueryBucket

    source
    pub struct QueryBucket {
      query_id : String
      label : String
      score : Double
    } derive(ToJson, 
    Debug    )

    QueryComparison

    source
    pub struct QueryComparison {
      query_id : String
      baseline : Double
      candidate : Double
      delta : Double
      outcome : String
      overlap : Int
    } derive(ToJson, 
    Debug    )

    QueryEvaluation

    source
    pub struct QueryEvaluation {
      query_id : String
      relevant_total : Int
      retrieved_total : Int
      metrics : Map[String, Double]
    } derive(ToJson, 
    Debug    )

    QueryPartition

    source
    pub struct QueryPartition {
      partition : Int
      query_ids : Array[String]
      qrels : Array[JudgedDoc]
    } derive(ToJson, 
    Debug    )

    QueryStatistic

    source
    pub struct QueryStatistic {
      query_id : String
      judged_count : Int
      relevant_count : Int
      retrieved_count : Int
      unjudged_count : Int
      mean_score : Double
    } derive(ToJson, 
    Debug    )

    RankPoint

    source
    pub struct RankPoint {
      rank : Int
      precision : Double
      recall : Double
    } derive(ToJson, 
    Debug    )

    RetrievedDoc

    source
    pub struct RetrievedDoc {
      query_id : String
      doc_id : String
      score : Double
    } derive(Eq, ToJson, 
    Debug    )

    RetrievedDoc::new

    source
    fn RetrievedDoc::new(query_id~ : String, doc_id~ : String, score~ : Double) -> RetrievedDoc

    RunComparison

    source
    pub struct RunComparison {
      cutoff : Int
      query_count : Int
      wins : Int
      losses : Int
      ties : Int
      mean_delta : Double
      queries : Array[QueryComparison]
    } derive(ToJson, 
    Debug    )

    RunOrder

    source
    pub enum RunOrder {
      ScoreDescending
      DocumentAscending
      InputOrder
    } derive(Eq, ToJson, 
    Debug    )

    RunOrder::document_ascending

    source
    fn RunOrder::document_ascending() -> RunOrder

    RunOrder::input_order

    source
    fn RunOrder::input_order() -> RunOrder

    RunOrder::score_descending

    source
    fn RunOrder::score_descending() -> RunOrder

    RunQuality

    source
    pub struct RunQuality {
      query_count : Int
      row_count : Int
      unique_row_count : Int
      duplicate_rate : Double
      unjudged_rate : Double
      score_monotonicity : Double
      finite_score_rate : Double
    } derive(ToJson, 
    Debug    )

    ThresholdEvaluation

    source
    pub struct ThresholdEvaluation {
      threshold : Int
      cutoff : Int
      precision : Double
      recall : Double
      f1 : Double
      ndcg : Double
    } derive(ToJson, 
    Debug    )

    TraceStep

    source
    pub struct TraceStep {
      rank : Int
      doc_id : String
      score : Double
      relevance : Int
      judged : Bool
      hits : Int
      precision : Double
      recall : Double
    } derive(ToJson, 
    Debug    )

    TsvScanStats

    source
    pub struct TsvScanStats {
      total_lines : Int
      blank_lines : Int
      comment_lines : Int
      data_lines : Int
    } derive(Eq, ToJson, 
    Debug    )

    ValidationIssue

    source
    pub struct ValidationIssue {
      code : String
      level : ValidationLevel
      message : String
      query_id : String
      doc_id : String
    } derive(Eq, ToJson, 
    Debug    )

    ValidationLevel

    source
    pub enum ValidationLevel {
      Warning
      Error
    } derive(Eq, ToJson, 
    Debug    )

    ValidationSummary

    source
    pub struct ValidationSummary {
      error_count : Int
      warning_count : Int
      query_count : Int
      judged_count : Int
      retrieved_count : Int
      issues : Array[ValidationIssue]
    } derive(ToJson, 
    Debug    )

    ValidationSummary::is_valid

    source
    fn ValidationSummary::is_valid(self : ValidationSummary) -> Bool

    API_VERSION

    source
    let API_VERSION : String

    DATA_SCHEMA_VERSION

    source
    let DATA_SCHEMA_VERSION : String

    api_healthcheck

    source
    fn api_healthcheck() -> String

    available_presets

    source
    fn available_presets() -> Array[String]

    average_precision_at

    source
    fn average_precision_at(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int) -> Double

    benchmark_case_is_reproducible

    source
    fn benchmark_case_is_reproducible(case : BenchmarkCase, cutoffs~ : Array[Int]) -> Bool

    benchmark_metric_table

    source
    fn benchmark_metric_table(result : BenchmarkResult) -> String

    best_query

    source
    fn best_query(report : BenchmarkReport, metric : String) -> String

    best_threshold

    source
    fn best_threshold(rows : Array[ThresholdEvaluation], cutoff : Int) -> Int

    blend_runs

    source
    fn blend_runs(left : Array[RetrievedDoc], right : Array[RetrievedDoc], left_weight? : Double) -> Array[RetrievedDoc]

    bottom_query_ids

    source
    fn bottom_query_ids(report : BenchmarkReport, metric : String, limit : Int) -> Array[String]

    bpref_at

    source
    fn bpref_at(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int) -> Double

    build_candidate_pools

    source
    fn build_candidate_pools(run : Array[RetrievedDoc]) -> Array[CandidatePool]

    bundle_is_reproducible

    source
    fn bundle_is_reproducible(case : BenchmarkCase, cutoffs~ : Array[Int]) -> Bool

    candidate_pool_coverage

    source
    fn candidate_pool_coverage(qrels : Array[JudgedDoc], pools : Array[CandidatePool]) -> Double

    candidate_recall

    source
    fn candidate_recall(qrels : Array[JudgedDoc], pools : Array[CandidatePool]) -> Double

    canonical_output_format

    source
    fn canonical_output_format(format : String) -> Result[CliFormat, String]

    classify_queries

    source
    fn classify_queries(report : BenchmarkReport, cutoff~ : Int) -> Array[QueryBucket]

    cli_format_name

    source
    fn cli_format_name(format : CliFormat) -> String

    cli_request_summary

    source
    fn cli_request_summary(request : CliRequest) -> String

    cli_usage_text

    source
    fn cli_usage_text() -> String

    compare_benchmark_cases

    source
    fn compare_benchmark_cases(baseline : BenchmarkCase, candidate : BenchmarkCase, cutoff~ : Int) -> RunComparison

    compare_metric_means

    source
    fn compare_metric_means(left : BenchmarkReport, right : BenchmarkReport, metric : String) -> Double

    compare_runs

    source
    fn compare_runs(qrels : Array[JudgedDoc], baseline : Array[RetrievedDoc], candidate : Array[RetrievedDoc], cutoff~ : Int) -> RunComparison

    config_signature

    source
    fn config_signature(config : EvalConfig) -> String

    corpus_profile

    source
    fn corpus_profile(qrels : Array[JudgedDoc], run : Array[RetrievedDoc]) -> CorpusProfile

    corpus_profile_json

    source
    fn corpus_profile_json(profile : CorpusProfile) -> String

    coverage_at

    source
    fn coverage_at(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int) -> Double

    deduplicate_candidate_pool

    source
    fn deduplicate_candidate_pool(pool : CandidatePool) -> CandidatePool

    deduplicate_qrels

    source
    fn deduplicate_qrels(qrels : Array[JudgedDoc]) -> Array[JudgedDoc]

    deduplicate_run

    source
    fn deduplicate_run(run : Array[RetrievedDoc]) -> Array[RetrievedDoc]

    default_cutoffs

    source
    fn default_cutoffs() -> Array[Int]

    demo_bundle

    source
    fn demo_bundle() -> String

    demo_fixture

    source
    fn demo_fixture() -> BenchmarkCase

    demo_fixture_pools

    source
    fn demo_fixture_pools() -> Array[CandidatePool]

    demo_pool_source

    source
    fn demo_pool_source() -> String

    demo_qrels_source

    source
    fn demo_qrels_source() -> String

    demo_report

    source
    fn demo_report() -> BenchmarkReport

    demo_run_source

    source
    fn demo_run_source() -> String

    demo_validation

    source
    fn demo_validation() -> ValidationSummary

    diagnose_run

    source
    fn diagnose_run(qrels : Array[JudgedDoc], run : Array[RetrievedDoc]) -> String

    document_relevance_levels

    source
    fn document_relevance_levels(qrels : Array[JudgedDoc]) -> Array[Int]

    document_statistics

    source
    fn document_statistics(qrels : Array[JudgedDoc]) -> Array[DocumentStatistic]

    err_at

    source
    fn err_at(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int) -> Double

    evaluate_benchmark

    source
    fn evaluate_benchmark(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], config? : EvalConfig) -> BenchmarkReport

    evaluate_metric_family

    source
    fn evaluate_metric_family(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int, gain_scheme : GainScheme) -> Map[String, Double]

    evaluate_query

    source
    fn evaluate_query(query_id : String, qrels : Array[JudgedDoc], run : Array[RetrievedDoc], config? : EvalConfig) -> QueryEvaluation

    evaluate_with_preset

    source
    fn evaluate_with_preset(name : String, qrels : Array[JudgedDoc], run : Array[RetrievedDoc]) -> Result[BenchmarkReport, String]

    example_eval_summary

    source
    fn example_eval_summary() -> String

    example_negative_sample_summary

    source
    fn example_negative_sample_summary() -> String

    example_profile_summary

    source
    fn example_profile_summary() -> String

    example_quality_summary

    source
    fn example_quality_summary() -> String

    execute_plan

    source
    fn execute_plan(plan : EvaluationPlan, qrels : Array[JudgedDoc], run : Array[RetrievedDoc]) -> PlanResult

    execute_plan_with_profile

    source
    fn execute_plan_with_profile(plan : EvaluationPlan, qrels : Array[JudgedDoc], run : Array[RetrievedDoc]) -> String

    export_bundle

    source
    fn export_bundle(case : BenchmarkCase, cutoffs~ : Array[Int]) -> String

    export_bundle_json

    source
    fn export_bundle_json(case : BenchmarkCase, cutoffs~ : Array[Int]) -> String

    export_report_files

    source
    fn export_report_files(report : BenchmarkReport) -> Map[String, String]

    export_report_index

    source
    fn export_report_index(report : BenchmarkReport) -> String

    f1_at

    source
    fn f1_at(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int) -> Double

    f1_from_counts

    source
    fn f1_from_counts(relevant_retrieved : Int, retrieved : Int, relevant : Int) -> Double

    fallout_at

    source
    fn fallout_at(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int) -> Double

    filter_qrels_by_queries

    source
    fn filter_qrels_by_queries(qrels : Array[JudgedDoc], query_ids : Array[String]) -> Array[JudgedDoc]

    filter_report_queries

    source
    fn filter_report_queries(report : BenchmarkReport, query_ids : Array[String]) -> BenchmarkReport

    filter_run_by_queries

    source
    fn filter_run_by_queries(run : Array[RetrievedDoc], query_ids : Array[String]) -> Array[RetrievedDoc]

    filter_run_by_score

    source
    fn filter_run_by_score(run : Array[RetrievedDoc], minimum_score : Double) -> Array[RetrievedDoc]

    first_relevant_rank

    source
    fn first_relevant_rank(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], threshold : Int) -> Int

    format_percent

    source
    fn format_percent(value : Double) -> String

    graded_precision_at

    source
    fn graded_precision_at(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int, gain_scheme : GainScheme) -> Double

    interleave_candidate_pools

    source
    fn interleave_candidate_pools(left : CandidatePool, right : CandidatePool) -> CandidatePool

    is_known_preset

    source
    fn is_known_preset(name : String) -> Bool

    is_supported_output_format

    source
    fn is_supported_output_format(format : String) -> Bool

    jaccard_at

    source
    fn jaccard_at(left : Array[RetrievedDoc], right : Array[RetrievedDoc], query_id : String, cutoff : Int) -> Double

    join_qrels_with_pool

    source
    fn join_qrels_with_pool(qrels : Array[JudgedDoc], pools : Array[CandidatePool]) -> Array[JudgedDoc]

    judged_recall_at

    source
    fn judged_recall_at(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int) -> Double

    manifest_with_pools

    source
    fn manifest_with_pools(name~ : String, version~ : String, license~ : String, qrels~ : Array[JudgedDoc], run~ : Array[RetrievedDoc], pools~ : Array[CandidatePool]) -> BenchmarkManifest

    merge_candidate_pools

    source
    fn merge_candidate_pools(left : Array[CandidatePool], right : Array[CandidatePool]) -> Array[CandidatePool]

    merge_runs

    source
    fn merge_runs(left : Array[RetrievedDoc], right : Array[RetrievedDoc]) -> Array[RetrievedDoc]

    metric_aliases

    source
    fn metric_aliases() -> Map[String, String]

    metric_mean_by_query

    source
    fn metric_mean_by_query(report : BenchmarkReport, prefix : String, cutoff : Int) -> Double

    metric_rank

    source
    fn metric_rank(report : BenchmarkReport, metric : String, query_id : String) -> Int

    metric_rank_delta

    source
    fn metric_rank_delta(baseline : BenchmarkReport, candidate : BenchmarkReport, metric : String, query_id : String) -> Int

    metric_wins

    source
    fn metric_wins(baseline : BenchmarkReport, candidate : BenchmarkReport, name : String) -> (Int, Int, Int)

    module_identifier

    source
    fn module_identifier() -> String

    ndcg_at

    source
    fn ndcg_at(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int, gain_scheme : GainScheme) -> Double

    normalize_cutoffs

    source
    fn normalize_cutoffs(cutoffs : Array[Int]) -> Array[Int]

    normalize_inputs

    source
    fn normalize_inputs(qrels : Array[JudgedDoc], run : Array[RetrievedDoc]) -> (Array[JudgedDoc], Array[RetrievedDoc])

    normalize_plan

    source
    fn normalize_plan(plan : EvaluationPlan) -> EvaluationPlan

    normalize_run

    source
    fn normalize_run(run : Array[RetrievedDoc]) -> Array[RetrievedDoc]

    normalize_scores

    source
    fn normalize_scores(run : Array[RetrievedDoc]) -> Array[RetrievedDoc]

    overlap_at

    source
    fn overlap_at(baseline : Array[RetrievedDoc], candidate : Array[RetrievedDoc], query_id : String, cutoff : Int) -> Int

    parse_candidate_pool_tsv

    source
    fn parse_candidate_pool_tsv(source : String) -> Result[Array[CandidatePool], String]

    parse_cli_request

    source
    fn parse_cli_request(args : Array[String]) -> Result[CliRequest, String]

    parse_qrels_tsv

    source
    fn parse_qrels_tsv(source : String) -> Result[Array[JudgedDoc], String]

    parse_qrels_tsv_strict

    source
    fn parse_qrels_tsv_strict(source : String) -> Result[Array[JudgedDoc], String]

    parse_run_tsv

    source
    fn parse_run_tsv(source : String) -> Result[Array[RetrievedDoc], String]

    parse_run_tsv_strict

    source
    fn parse_run_tsv_strict(source : String) -> Result[Array[RetrievedDoc], String]

    partition_qrels

    source
    fn partition_qrels(qrels : Array[JudgedDoc], partition_count : Int) -> Array[QueryPartition]

    partition_row_count

    source
    fn partition_row_count(partitions : Array[QueryPartition]) -> Int

    plan_has_metric

    source
    fn plan_has_metric(result : PlanResult, name : String) -> Bool

    plan_metric

    source
    fn plan_metric(result : PlanResult, name : String) -> MetricDistribution

    plan_sampling

    source
    fn plan_sampling(qrels : Array[JudgedDoc], pools : Array[CandidatePool], total_budget~ : Int) -> String

    plan_to_json

    source
    fn plan_to_json(result : PlanResult) -> String

    plan_validation_markdown

    source
    fn plan_validation_markdown(result : PlanResult) -> String

    precision_at

    source
    fn precision_at(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int) -> Double

    precision_curve

    source
    fn precision_curve(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int) -> Array[Double]

    precision_from_counts

    source
    fn precision_from_counts(relevant_retrieved : Int, retrieved : Int) -> Double

    precision_recall_area

    source
    fn precision_recall_area(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int) -> Double

    precision_recall_points

    source
    fn precision_recall_points(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int) -> Array[RankPoint]

    preset_config

    source
    fn preset_config(name : String) -> Result[EvalConfig, String]

    preset_cutoffs

    source
    fn preset_cutoffs(name : String) -> Result[Array[Int], String]

    preset_description

    source
    fn preset_description(name : String) -> Result[String, String]

    profile_candidate_pools

    source
    fn profile_candidate_pools(pools : Array[CandidatePool]) -> CandidatePoolProfile

    profile_dataset

    source
    fn profile_dataset(qrels : Array[JudgedDoc], run : Array[RetrievedDoc]) -> DatasetProfile

    quality_gate

    source
    fn quality_gate(case : BenchmarkCase, report : BenchmarkReport) -> String

    quantile

    source
    fn quantile(values : Array[Double], fraction : Double) -> Double

    query_metric_rows

    source
    fn query_metric_rows(report : BenchmarkReport, metric_names : Array[String]) -> String

    query_statistics

    source
    fn query_statistics(qrels : Array[JudgedDoc], run : Array[RetrievedDoc]) -> Array[QueryStatistic]

    query_statistics_markdown

    source
    fn query_statistics_markdown(stats : Array[QueryStatistic]) -> String

    r_precision

    source
    fn r_precision(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], threshold : Int) -> Double

    rank_biased_overlap

    source
    fn rank_biased_overlap(left : Array[RetrievedDoc], right : Array[RetrievedDoc], query_id : String, cutoff : Int, persistence? : Double) -> Double

    rank_movement

    source
    fn rank_movement(baseline : Array[RetrievedDoc], candidate : Array[RetrievedDoc], query_id : String, cutoff : Int) -> Int

    rbp_at

    source
    fn rbp_at(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int, persistence? : Double) -> Double

    recall_at

    source
    fn recall_at(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int) -> Double

    recall_curve

    source
    fn recall_curve(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int) -> Array[Double]

    recall_from_counts

    source
    fn recall_from_counts(relevant_retrieved : Int, relevant : Int) -> Double

    reciprocal_rank_at

    source
    fn reciprocal_rank_at(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff : Int, threshold : Int) -> Double

    release_channel

    source
    fn release_channel() -> String

    release_checklist

    source
    fn release_checklist() -> Array[String]

    release_metadata

    source
    fn release_metadata() -> String

    release_notes

    source
    fn release_notes() -> String

    release_target_count

    source
    fn release_target_count() -> Int

    release_target_names

    source
    fn release_target_names() -> Array[String]

    relevance_histogram

    source
    fn relevance_histogram(qrels : Array[JudgedDoc]) -> Map[Int, Int]

    relevant_rank_positions

    source
    fn relevant_rank_positions(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], threshold : Int) -> Array[Int]

    render_benchmark_manifest

    source
    fn render_benchmark_manifest(case : BenchmarkCase) -> String

    render_benchmark_result

    source
    fn render_benchmark_result(result : BenchmarkResult) -> String

    render_compact_report

    source
    fn render_compact_report(report : BenchmarkReport) -> String

    render_comparison_json

    source
    fn render_comparison_json(comparison : RunComparison) -> String

    render_comparison_markdown

    source
    fn render_comparison_markdown(comparison : RunComparison) -> String

    render_comparison_text

    source
    fn render_comparison_text(comparison : RunComparison) -> String

    render_corpus_profile

    source
    fn render_corpus_profile(profile : CorpusProfile) -> String

    render_csv_report

    source
    fn render_csv_report(report : BenchmarkReport) -> String

    render_distribution_markdown

    source
    fn render_distribution_markdown(name : String, distribution : MetricDistribution) -> String

    render_histogram

    source
    fn render_histogram(histogram : Map[Int, Int]) -> String

    render_html_report

    source
    fn render_html_report(report : BenchmarkReport) -> String

    render_json_lines_report

    source
    fn render_json_lines_report(report : BenchmarkReport) -> String

    render_json_report

    source
    fn render_json_report(report : BenchmarkReport) -> String

    render_manifest

    source
    fn render_manifest(manifest : BenchmarkManifest) -> String

    render_manifest_json

    source
    fn render_manifest_json(manifest : BenchmarkManifest) -> String

    render_markdown_report

    source
    fn render_markdown_report(report : BenchmarkReport) -> String

    render_metric_matrix

    source
    fn render_metric_matrix(reports : Array[(String, BenchmarkReport)]) -> String

    render_negative_samples_tsv

    source
    fn render_negative_samples_tsv(samples : Array[NegativeSample]) -> String

    render_plan_summary

    source
    fn render_plan_summary(result : PlanResult) -> String

    render_pools_tsv

    source
    fn render_pools_tsv(pools : Array[CandidatePool]) -> String

    render_preset_catalog

    source
    fn render_preset_catalog() -> String

    render_preset_report

    source
    fn render_preset_report(name : String, report : BenchmarkReport) -> String

    render_profile_json

    source
    fn render_profile_json(profile : DatasetProfile) -> String

    render_profile_text

    source
    fn render_profile_text(profile : DatasetProfile) -> String

    render_qrels_tsv

    source
    fn render_qrels_tsv(qrels : Array[JudgedDoc]) -> String

    render_quality_summary

    source
    fn render_quality_summary(quality : RunQuality) -> String

    render_query_buckets_markdown

    source
    fn render_query_buckets_markdown(buckets : Array[QueryBucket]) -> String

    render_ranked_metric_table

    source
    fn render_ranked_metric_table(report : BenchmarkReport, metric : String) -> String

    render_reproducibility_card

    source
    fn render_reproducibility_card(case : BenchmarkCase) -> String

    render_run_tsv

    source
    fn render_run_tsv(run : Array[RetrievedDoc]) -> String

    render_statistics_markdown

    source
    fn render_statistics_markdown(qrels : Array[JudgedDoc], run : Array[RetrievedDoc]) -> String

    render_threshold_markdown

    source
    fn render_threshold_markdown(rows : Array[ThresholdEvaluation]) -> String

    render_threshold_sweep

    source
    fn render_threshold_sweep(rows : Array[ThresholdEvaluation]) -> String

    render_trace_tsv

    source
    fn render_trace_tsv(steps : Array[TraceStep]) -> String

    render_validation_and_profile

    source
    fn render_validation_and_profile(qrels : Array[JudgedDoc], run : Array[RetrievedDoc]) -> String

    render_validation_report

    source
    fn render_validation_report(summary : ValidationSummary) -> String

    report_digest

    source
    fn report_digest(report : BenchmarkReport) -> String

    report_has_finite_metrics

    source
    fn report_has_finite_metrics(report : BenchmarkReport) -> Bool

    report_metric_names

    source
    fn report_metric_names(report : BenchmarkReport) -> Array[String]

    resolve_metric_name

    source
    fn resolve_metric_name(prefix : String, cutoff : Int) -> String

    retrieved_document_count

    source
    fn retrieved_document_count(run : Array[RetrievedDoc]) -> Int

    run_benchmark_case

    source
    fn run_benchmark_case(case : BenchmarkCase, cutoffs~ : Array[Int]) -> BenchmarkResult

    run_quality

    source
    fn run_quality(qrels : Array[JudgedDoc], run : Array[RetrievedDoc]) -> RunQuality

    run_signature

    source
    fn run_signature(run : Array[RetrievedDoc]) -> String

    run_to_pool

    source
    fn run_to_pool(run : Array[RetrievedDoc], query_id : String) -> CandidatePool

    safe_compare

    source
    fn safe_compare(qrels : Array[JudgedDoc], baseline : Array[RetrievedDoc], candidate : Array[RetrievedDoc]) -> Result[RunComparison, String]

    safe_evaluate

    source
    fn safe_evaluate(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], config? : EvalConfig) -> Result[BenchmarkReport, String]

    safe_parse_and_evaluate

    source
    fn safe_parse_and_evaluate(qrels_source : String, run_source : String, config : EvalConfig) -> Result[BenchmarkReport, String]

    safe_report_json

    source
    fn safe_report_json(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], config? : EvalConfig) -> Result[String, String]

    safe_sample

    source
    fn safe_sample(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], config : NegativeSampleConfig) -> Result[Array[NegativeSample], String]

    sample_hard_and_tail

    source
    fn sample_hard_and_tail(qrels : Array[JudgedDoc], pools : Array[CandidatePool], per_query~ : Int, window~ : Int) -> Array[NegativeSample]

    sample_negatives

    source
    fn sample_negatives(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], pools? : Array[CandidatePool], config? : NegativeSampleConfig) -> Array[NegativeSample]

    sample_pool_negatives

    source
    fn sample_pool_negatives(qrels : Array[JudgedDoc], pools : Array[CandidatePool], config? : NegativeSampleConfig) -> Array[NegativeSample]

    sample_pool_negatives_tsv

    source
    fn sample_pool_negatives_tsv(qrels : Array[JudgedDoc], pools : Array[CandidatePool], config? : NegativeSampleConfig) -> String

    sample_with_budget

    source
    fn sample_with_budget(qrels : Array[JudgedDoc], pools : Array[CandidatePool], total_budget~ : Int) -> Array[NegativeSample]

    sampling_plan

    source
    fn sampling_plan(qrels : Array[JudgedDoc], pools : Array[CandidatePool], config? : NegativeSampleConfig) -> String

    sampling_strategy_description

    source
    fn sampling_strategy_description(strategy : NegativeStrategy) -> String

    sampling_yield

    source
    fn sampling_yield(qrels : Array[JudgedDoc], pools : Array[CandidatePool], samples : Array[NegativeSample]) -> Double

    scan_tsv_source

    source
    fn scan_tsv_source(source : String) -> TsvScanStats

    score_bucket_counts

    source
    fn score_bucket_counts(run : Array[RetrievedDoc], bucket_count : Int) -> Array[Int]

    score_mean

    source
    fn score_mean(run : Array[RetrievedDoc]) -> Double

    score_monotonicity

    source
    fn score_monotonicity(run : Array[RetrievedDoc]) -> Double

    score_order_correlation

    source
    fn score_order_correlation(left : Array[RetrievedDoc], right : Array[RetrievedDoc], query_id : String, cutoff : Int) -> Double

    score_stddev

    source
    fn score_stddev(run : Array[RetrievedDoc]) -> Double

    select_queries_by_score

    source
    fn select_queries_by_score(report : BenchmarkReport, metric : String, minimum : Double) -> Array[String]

    sort_run

    source
    fn sort_run(run : Array[RetrievedDoc], order : RunOrder) -> Array[RetrievedDoc]

    summarize_report_metric

    source
    fn summarize_report_metric(report : BenchmarkReport, name : String) -> MetricDistribution

    summarize_values

    source
    fn summarize_values(values : Array[Double]) -> MetricDistribution

    supported_output_formats

    source
    fn supported_output_formats() -> Array[String]

    threshold_frontier

    source
    fn threshold_frontier(rows : Array[ThresholdEvaluation]) -> Array[ThresholdEvaluation]

    threshold_sweep

    source
    fn threshold_sweep(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], thresholds~ : Array[Int], cutoffs~ : Array[Int]) -> Array[ThresholdEvaluation]

    top_query_ids

    source
    fn top_query_ids(report : BenchmarkReport, metric : String, limit : Int) -> Array[String]

    trace_auc

    source
    fn trace_auc(steps : Array[TraceStep]) -> Double

    trace_first_judged_rank

    source
    fn trace_first_judged_rank(steps : Array[TraceStep]) -> Int

    trace_first_relevant_rank

    source
    fn trace_first_relevant_rank(steps : Array[TraceStep]) -> Int

    trace_query

    source
    fn trace_query(qrels : Array[JudgedDoc], run : Array[RetrievedDoc], cutoff~ : Int, threshold~ : Int) -> Array[TraceStep]

    trimmed_mean

    source
    fn trimmed_mean(values : Array[Double], trim_fraction : Double) -> Double

    truncate_run

    source
    fn truncate_run(run : Array[RetrievedDoc], per_query : Int) -> Array[RetrievedDoc]

    unique_documents_at

    source
    fn unique_documents_at(run : Array[RetrievedDoc], query_id : String, cutoff : Int) -> Array[String]

    validate_dataset

    source
    fn validate_dataset(qrels : Array[JudgedDoc], run : Array[RetrievedDoc]) -> ValidationSummary

    validate_eval_config

    source
    fn validate_eval_config(config : EvalConfig) -> Result[Unit, String]

    validate_manifest

    source
    fn validate_manifest(manifest : BenchmarkManifest) -> ValidationSummary

    weighted_mean

    source
    fn weighted_mean(values : Array[Double], weights : Array[Double]) -> Double

    worst_query

    source
    fn worst_query(report : BenchmarkReport, metric : String) -> String

    Source Files

    Powered by MoonBit

    Site sourceReport issueModulesSkillsCLIBuild queueStatistics

    © 2026 mooncakes.io