🥮 mooncakes.io

    moonalign

    A MoonBit-native bilingual paragraph and sentence alignment toolkit with dynamic-programming based matching and a practical CLI.

    alignment
    bilingual
    nlp
    translation
    corpus
    Download zip
    Author
    Version
    0.2.2
    License
    Apache-2.0
    Repository
    Last updated
    last month
    Downloads
    20

    #MoonAlign

    MoonAlign 是一个使用 MoonBit 编写的双语文本对齐工具包，用于把段落或句子级的源文本与目标文本转换为可审查、可评估、可继续处理的对齐结果。

    它适合用于平行语料构建、翻译流程清洗、术语对照、双语文档质量检查，以及需要轻量、可解释对齐能力的文本工程工具。项目提供 MoonBit 库 API 和命令行界面，不依赖在线服务或大型模型即可运行。

    #项目定位

    MoonAlign 关注“稳定的对齐底座”，而不是翻译或语义质量的黑盒替代品。它使用透明的长度特征、词元特征和动态规划搜索，在保持可解释性的同时处理常见的 1-1、1-2、2-1 对齐关系。

    核心设计目标：

    • MoonBit 原生实现，便于嵌入其他 MoonBit 项目。
    • 离线运行，结果稳定，可在 CI 和数据流水线中复现。
    • 输出结构化报告，保留跨度、得分、比例、告警和质量指标。
    • 通过词法锚点、质量门禁和审校队列支持人工复核。

    #核心能力

    • 句子级和段落级切分，兼容中英文标点、Unicode 空白和 CRLF 文本。
    • 基于动态规划的有限扇出对齐，支持 1-1、1-2、2-1 等合并路径。
    • ASCII 词元、CJK 字符和标点的轻量加权长度模型。
    • URL、数字、标识符和共享词元的词法锚点提取。
    • 单文档和批量语料 API，支持输入校验、批量对齐和结果汇总。
    • 质量评估：覆盖率、置信度、单调性、合并比例、告警和质量门禁。
    • 人工审校队列，以及 JSON、TSV、CSV 和特征向量导出。
    • 可解释的对齐结构特征和文本单元特征，可用于排序、质量校准和数据分析。

    #快速开始

    #环境要求

    • MoonBit CLI stable
    • Git

    在项目根目录运行：

    moon check
moon test

    #库 API

    ///|
    test "basic alignment" {
      let report = @moonalign.align(
        "MoonBit favors maintainable tooling.\n", "MoonBit 强调可维护的工具链。\n",
      )
      inspect(report.pairs.length(), content="1")
      inspect(report.pairs[0].move_kind, content="1-1")
    }

    批量处理和质量审校：

    ///|
    let documents = @moonalign.synthetic_corpus(10)

    ///|
    let results = @moonalign.align_corpus(documents)

    ///|
    let summary = @moonalign.summarize_corpus(results)

    ///|
    let review = @moonalign.make_review_queue(results[0])

    #安装为依赖

    项目模块名为 LL728/moonalign，版本和仓库信息位于 moon.mod。在其他 MoonBit 项目中添加依赖后，可通过 @moonalign 使用公开 API。

    #CLI

    MoonAlign CLI 位于 cmd/main，支持直接传入文本并输出 JSON 或 TSV：

    moon run cmd/main -- \
  --source-text "MoonBit favors stable tooling. MoonAlign builds corpora." \
  --target-text "MoonBit 强调稳定工具链。MoonAlign 用来构建语料。" \
  --mode sentence \
  --format json

    常用参数：

    参数说明
    --source-text <text>源语言文本
    --target-text <text>目标语言文本
    --mode sentence\|paragraph句子级或段落级切分，默认 sentence
    --format json\|tsvJSON 或 TSV 输出，默认 json
    --benchmark运行内置离线基准并输出 JSON 指标
    --help显示帮助

    输出结果包含源/目标跨度、移动类型、文本内容、长度权重、逐步得分和诊断告警。

    运行基准：

    moon run --target wasm-gc cmd/main -- --benchmark

    #架构

    source / target text
        │
        ▼
normalization and segmentation
        │
        ▼
weighted text units ─────── lexical anchors
        │                            │
        └──────── dynamic programming alignment
                                      │
                                      ▼
                         alignment report and metrics
                                      │
                   ┌──────────────────┼──────────────────┐
                   ▼                  ▼                  ▼
                 JSON/TSV          quality gate       review queue

    主要模块：

    • normalize.mbt：空白归一化、段落识别、句子切分和文本权重计算。
    • aligner.mbt：有限扇出动态规划、路径回溯和对齐跨度生成。
    • anchors.mbt：共享词元、URL、数字和标识符锚点。
    • metrics.mbt、quality.mbt：gold set 评估和质量门禁。
    • corpus.mbt、review.mbt：批量处理、导出和人工审校工作流。
    • feature_engineering.mbt：用于排序和校准的确定性特征向量。
    • cmd/main/main.mbt：命令行参数解析和输出格式选择。

    #基准

    仓库包含可离线运行的真实小样本基准，数据文件位于 benchmarks/tatoeba-eng-cmn.tsv，并保留 Tatoeba sentence ID、来源链接和许可证说明。

    当前基准输出包括：

    数据集PrecisionRecallF1覆盖率
    tatoeba-eng-cmn-short0.25000.22220.23531.0000
    tatoeba-eng-cmn-merge1.00001.00001.00001.0000

    第一个样本专门暴露长度启发式在短句和中英文长度差异下的边界；第二个样本用于保证 1-2 合并路径持续可用。基准不是翻译质量分数，也不宣称代表大规模语料分布。

    更多来源和数据许可信息见 benchmarks/README.md 与 SOURCES.md。

    #测试

    测试覆盖基础对齐、切分、归一化、gold set 指标、批量 API、质量门禁、特征向量、CLI smoke test 和边界输入。

    本地推荐执行：

    moon fmt --check
moon check --target all --deny-warn
moon test --target wasm-gc --deny-warn
moon test --target native --deny-warn
moon info

    边界回归测试覆盖空文本、空白、CRLF、Unicode、中文/英文混合、数字、URL、标点、段落空行、极端长度比和合并边界。

    #CI

    GitHub Actions 工作流位于 .github/workflows/check.yml，在 ubuntu-latest、macos-latest 和 windows-latest 上运行，并执行：

    • stable MoonBit CLI 安装和工具链同步；
    • 格式检查、全后端类型检查和 warning 拒绝；
    • wasm-gc 与 native 测试；
    • 公共接口生成和 git diff 漂移检查；
    • 离线基准回归和 CLI smoke test；
    • 边界测试数量检查。

    #许可证

    MoonAlign 使用 Apache License 2.0 发布。仓库中的 Tatoeba 基准数据遵循其来源许可证并保留必要署名；使用或再分发该数据时，请同时阅读 SOURCES.md 和 benchmarks/README.md。

    #相关文档

    AlignOptions

    source
    pub struct AlignOptions {
      segment_mode : SegmentMode
      max_fan_out : Int
      join_penalty : Double
      deviation_penalty : Double
      min_sentence_chars : Int
      preserve_paragraphs : Bool
    } derive(
    Debug    )

    AlignmentMetrics

    source
    pub struct AlignmentMetrics {
      predicted_pairs : Int
      gold_pairs : Int
      exact_pairs : Int
      precision : Double
      recall : Double
      f1 : Double
      source_coverage : Double
      target_coverage : Double
      merged_pairs : Int
      average_score : Double
    } derive(ToJson, 
    Debug    )

    AlignmentPair

    source
    pub struct AlignmentPair {
      source_start : Int
      source_end : Int
      target_start : Int
      target_end : Int
      source_text : String
      target_text : String
      source_char_weight : Double
      target_char_weight : Double
      source_tokens : Int
      target_tokens : Int
      score : Double
      move_kind : String
    } derive(ToJson, 
    Debug    )

    AlignmentQuality

    source
    pub struct AlignmentQuality {
      source_units : Int
      target_units : Int
      aligned_pairs : Int
      exact_one_to_one : Int
      merged_pairs : Int
      anchored_pairs : Int
      source_coverage : Double
      target_coverage : Double
      mean_confidence : Double
      monotonicity : Double
      score : Double
      issues : Array[QualityIssue]
    } derive(ToJson, 
    Debug    )

    Quality statistics suitable for CI regression checks.

    AlignmentReport

    source
    pub struct AlignmentReport {
      options : Json
      source_count : Int
      target_count : Int
      estimated_ratio : Double
      warnings : Array[String]
      pairs : Array[AlignmentPair]
    } derive(ToJson, 
    Debug    )

    AnchorOptions

    source
    pub struct AnchorOptions {
      min_token_length : Int
      include_numbers : Bool
      include_urls : Bool
      include_identifiers : Bool
      case_sensitive : Bool
      max_anchors_per_unit : Int
    } derive(
    Debug    )

    A configurable set of deterministic anchor extraction rules.

    BackPointer

    source
    type BackPointer derive(Eq, 
    Debug    )

    BenchmarkCase

    source
    pub struct BenchmarkCase {
      name : String
      source : String
      target : String
      gold_pairs : Array[GoldPair]
      source_url : String
      license : String
    } derive(ToJson, 
    Debug    )

    BenchmarkResult

    source
    pub struct BenchmarkResult {
      name : String
      metrics : AlignmentMetrics
      warnings : Array[String]
    } derive(ToJson, 
    Debug    )

    CorpusDocument

    source
    pub struct CorpusDocument {
      id : String
      source : String
      target : String
      metadata : Map[String, String]
    } derive(
    Debug    )

    A named bilingual document in a batch.

    CorpusDocument::with_metadata

    source
    fn CorpusDocument::with_metadata(document : CorpusDocument, key~ : String, value~ : String) -> CorpusDocument

    Add or replace a metadata field on a corpus document.

    CorpusResult

    source
    pub struct CorpusResult {
      id : String
      report : AlignmentReport
      quality : AlignmentQuality
    } derive(ToJson, 
    Debug    )

    A batch alignment result with aggregate quality information.

    CorpusSummary

    source
    pub struct CorpusSummary {
      document_count : Int
      total_source_units : Int
      total_target_units : Int
      total_pairs : Int
      mean_f1 : Double
      mean_quality_score : Double
      warning_count : Int
      issue_count : Int
    } derive(ToJson, 
    Debug    )

    Aggregate statistics over a batch run.

    GoldPair

    source
    pub struct GoldPair {
      source_start : Int
      source_end : Int
      target_start : Int
      target_end : Int
    } derive(Eq, ToJson, 
    Debug    )

    LengthStats

    source
    pub struct LengthStats {
      count : Int
      total : Double
      minimum : Double
      maximum : Double
      mean : Double
      median : Double
      p90 : Double
    } derive(ToJson, 
    Debug    )

    A transparent distribution of weighted text lengths.

    LexicalAnchor

    source
    pub struct LexicalAnchor {
      source_unit : Int
      target_unit : Int
      source_text : String
      target_text : String
      normalized : String
      kind : String
      score : Double
    } derive(ToJson, 
    Debug    )

    A lexical anchor found in both sides of a bilingual document.

    Anchors are intentionally lightweight: they are useful for terminology, identifiers, URLs, numbers, and names without requiring a language model.

    QualityGate

    source
    pub struct QualityGate {
      min_source_coverage : Double
      min_target_coverage : Double
      min_mean_confidence : Double
      min_quality_score : Double
      max_warning_count : Int
    } derive(
    Debug    )

    Thresholds used by automated quality gates.

    QualityIssue

    source
    pub struct QualityIssue {
      code : String
      severity : String
      message : String
      source_unit : Int?
      target_unit : Int?
    } derive(ToJson, 
    Debug    )

    A document-level quality diagnostic.

    ReviewItem

    source
    pub struct ReviewItem {
      document_id : String
      pair_index : Int
      label : ReviewLabel
      confidence : Double
      reason : String
      anchors : Int
    } derive(ToJson, 
    Debug    )

    A pair plus the evidence used to make a review decision.

    ReviewLabel

    source
    pub enum ReviewLabel {
      Accept
      NeedsReview
      Reject
    } derive(Eq, ToJson, 
    Debug    )

    A review label assigned to an alignment pair.

    ReviewQueue

    source
    pub struct ReviewQueue {
      items : Array[ReviewItem]
      accepted : Int
      needs_review : Int
      rejected : Int
    } derive(ToJson, 
    Debug    )

    An ordered human-review queue.

    SegmentMode

    source
    pub enum SegmentMode {
      Paragraph
      Sentence
    } derive(Eq, 
    Debug    )

    TextUnit

    source
    pub struct TextUnit {
      id : Int
      paragraph_index : Int
      sentence_index : Int
      text : String
      normalized : String
      char_weight : Double
      token_count : Int
    } derive(ToJson, 
    Debug    )

    accepted_documents

    source
    fn accepted_documents(results : Array[CorpusResult], gate? : QualityGate) -> Array[CorpusResult]

    Filter batch results to documents that pass a quality gate.

    align

    source
    fn align(source_text : String, target_text : String, options? : AlignOptions) -> AlignmentReport

    align_corpus

    source
    fn align_corpus(documents : Array[CorpusDocument], options? : AlignOptions) -> Array[CorpusResult]

    Align every document in a batch while preserving input order.

    align_document

    source
    fn align_document(document : CorpusDocument, options? : AlignOptions) -> CorpusResult

    align_texts

    source
    fn align_texts(source_text : String, target_text : String, options? : AlignOptions) -> AlignmentReport

    align_units

    source
    fn align_units(source_units : Array[TextUnit], target_units : Array[TextUnit], options? : AlignOptions) -> AlignmentReport

    align_with_mode

    source
    fn align_with_mode(source_text : String, target_text : String, mode? : SegmentMode) -> AlignmentReport

    alignment_feature_mean

    source
    fn alignment_feature_mean(pair : AlignmentPair) -> Double

    Return the mean value of the structural alignment features.

    alignment_feature_names

    source
    fn alignment_feature_names() -> Array[String]

    Names for the interpretable alignment feature vector.

    alignment_feature_vector

    source
    fn alignment_feature_vector(pair : AlignmentPair) -> Array[Double]

    Extract interpretable structural and scoring features from one pair.

    anchor_bonus

    source
    fn anchor_bonus(pair : AlignmentPair, anchors : Array[LexicalAnchor]) -> Double

    Compute an anchor bonus for a proposed alignment pair.

    anchor_kind_counts

    source
    fn anchor_kind_counts(anchors : Array[LexicalAnchor]) -> Map[String, Int]

    Count anchors grouped by their category.

    assess_quality

    source
    fn assess_quality(report : AlignmentReport, anchors? : Array[LexicalAnchor]) -> AlignmentQuality

    Build a quality report from alignment output and lexical evidence.

    benchmark_case

    source
    fn benchmark_case(case : BenchmarkCase) -> BenchmarkResult

    benchmark_cases

    source
    fn benchmark_cases() -> Array[BenchmarkCase]

    benchmark_suite

    source
    fn benchmark_suite() -> Array[BenchmarkResult]

    benchmark_to_json_string

    source
    fn benchmark_to_json_string(indent? : Int) -> String

    corpus_document

    source
    fn corpus_document(id~ : String, source~ : String, target~ : String) -> CorpusDocument

    Create a corpus document without requiring a mutable metadata map.

    corpus_quality_csv

    source
    fn corpus_quality_csv(results : Array[CorpusResult]) -> String

    Render a compact quality dashboard as CSV.

    corpus_to_tsv

    source
    fn corpus_to_tsv(results : Array[CorpusResult]) -> String

    Render aligned corpus pairs in a review-friendly TSV format.

    default_anchor_options

    source
    fn default_anchor_options() -> AnchorOptions

    Return conservative defaults for lexical anchor extraction.

    default_options

    source
    fn default_options() -> AlignOptions

    default_quality_gate

    source
    fn default_quality_gate() -> QualityGate

    Return thresholds appropriate for human review queues.

    evaluate

    source
    fn evaluate(report : AlignmentReport, gold_pairs : Array[GoldPair]) -> AlignmentMetrics

    extract_anchors

    source
    fn extract_anchors(units : Array[TextUnit], options? : AnchorOptions) -> Array[LexicalAnchor]

    Extract deterministic anchors from segmented units.

    feature_vector_to_csv

    source
    fn feature_vector_to_csv(values : Array[Double]) -> String

    Convert a named feature vector into a stable CSV row.

    gold_pair

    source
    fn gold_pair(source_start~ : Int, source_end~ : Int, target_start~ : Int, target_end~ : Int) -> GoldPair

    length_ratio

    source
    fn length_ratio(source : Array[TextUnit], target : Array[TextUnit]) -> Double

    Compute a weighted length ratio for two segmented documents.

    length_stats

    source
    fn length_stats(units : Array[TextUnit]) -> LengthStats

    Compute min/max/mean/median/p90 from weighted units.

    make_review_queue

    source
    fn make_review_queue(result : CorpusResult) -> ReviewQueue

    Build an ordered review queue from one document result.

    make_review_queues

    source
    fn make_review_queues(results : Array[CorpusResult]) -> Array[ReviewQueue]

    Build queues for a complete batch.

    match_anchors

    source
    fn match_anchors(source_units : Array[TextUnit], target_units : Array[TextUnit], options? : AnchorOptions) -> Array[LexicalAnchor]

    Create cross-language anchor candidates using shared normalized tokens.

    move_kind_counts

    source
    fn move_kind_counts(results : Array[CorpusResult]) -> Map[String, Int]

    Count pairs by move shape for regression monitoring.

    named_feature_vector_to_csv

    source
    fn named_feature_vector_to_csv(names : Array[String], values : Array[Double]) -> String

    Render the feature schema and values as a reviewable CSV record.

    normalize_text

    source
    fn normalize_text(text : String) -> String

    options_to_json

    source
    fn options_to_json(options : AlignOptions) -> Json

    paragraph_histogram

    source
    fn paragraph_histogram(units : Array[TextUnit]) -> Map[Int, Int]

    Count units by paragraph while keeping empty paragraphs visible in reports.

    paragraph_mode

    source
    fn paragraph_mode() -> SegmentMode

    parse_corpus_tsv

    source
    fn parse_corpus_tsv(text : String) -> Array[CorpusDocument]

    Parse a minimal tab-separated corpus input. Blank and comment lines skip.

    passes_quality_gate

    source
    fn passes_quality_gate(quality : AlignmentQuality, gate? : QualityGate) -> Bool

    Check whether a quality result is acceptable for automated ingestion.

    pending_review_items

    source
    fn pending_review_items(queue : ReviewQueue) -> Array[ReviewItem]

    Return only items which should be shown to a human.

    quality_issue

    source
    fn quality_issue(code~ : String, severity~ : String, message~ : String, source_unit? : Int, target_unit? : Int) -> QualityIssue

    Create a quality issue with optional unit locations.

    quality_recommendations

    source
    fn quality_recommendations(quality : AlignmentQuality) -> Array[String]

    Produce human-readable review actions from quality diagnostics.

    quality_to_csv_row

    source
    fn quality_to_csv_row(id : String, quality : AlignmentQuality) -> String

    Return a stable CSV row for quality dashboards.

    repeated_units

    source
    fn repeated_units(units : Array[TextUnit]) -> Map[String, Int]

    Return all repeated normalized units and their frequency.

    report_to_json_string

    source
    fn report_to_json_string(report : AlignmentReport, indent? : Int) -> String

    report_to_tsv

    source
    fn report_to_tsv(report : AlignmentReport) -> String

    review_label_counts

    source
    fn review_label_counts(queues : Array[ReviewQueue]) -> Map[String, Int]

    Count review labels across a batch.

    review_queue

    source
    fn review_queue(results : Array[CorpusResult]) -> Array[CorpusResult]

    Filter batch results to documents requiring human review.

    review_queue_to_tsv

    source
    fn review_queue_to_tsv(queue : ReviewQueue) -> String

    Render a review queue as TSV for spreadsheet workflows.

    review_summary

    source
    fn review_summary(queue : ReviewQueue) -> String

    Return a stable summary sentence for a review dashboard.

    segment_text

    source
    fn segment_text(text : String, options? : AlignOptions) -> Array[TextUnit]

    sentence_mode

    source
    fn sentence_mode() -> SegmentMode

    short_unit_indexes

    source
    fn short_unit_indexes(units : Array[TextUnit], threshold? : Double) -> Array[Int]

    Return all unit indexes whose normalized text is unusually short.

    stats_to_csv_row

    source
    fn stats_to_csv_row(label : String, stats : LengthStats) -> String

    Serialize basic statistics as one stable CSV row.

    summarize_corpus

    source
    fn summarize_corpus(results : Array[CorpusResult]) -> CorpusSummary

    Summarize batch output for CI and release notes.

    synthetic_corpus

    source
    fn synthetic_corpus(count : Int) -> Array[CorpusDocument]

    Build a deterministic synthetic corpus used by performance tests.

    text_feature_names

    source
    fn text_feature_names() -> Array[String]

    Names for the interpretable text-unit feature vector.

    text_feature_peak

    source
    fn text_feature_peak(unit : TextUnit) -> Double

    Return the largest lexical/layout feature for a text unit.

    text_feature_vector

    source
    fn text_feature_vector(unit : TextUnit) -> Array[Double]

    Extract lexical, layout, and position features from one text unit.

    text_profile

    source
    fn text_profile(text : String, options? : AlignOptions) -> String

    Produce a deterministic text profile used in issue reports.

    token_overlap

    source
    fn token_overlap(left : TextUnit, right : TextUnit) -> Double

    Return normalized token overlap between two units.

    validate_corpus

    source
    fn validate_corpus(documents : Array[CorpusDocument]) -> Array[String]

    Validate a batch before running expensive alignment.

    with_max_fan_out

    source
    fn with_max_fan_out(options : AlignOptions, max_fan_out : Int) -> AlignOptions

    with_min_sentence_chars

    source
    fn with_min_sentence_chars(options : AlignOptions, min_sentence_chars : Int) -> AlignOptions

    with_preserve_paragraphs

    source
    fn with_preserve_paragraphs(options : AlignOptions, preserve : Bool) -> AlignOptions

    with_segment_mode

    source
    fn with_segment_mode(options : AlignOptions, mode : SegmentMode) -> AlignOptions

    Source Files

    Powered by MoonBit

    Site sourceReport issuePackagesBuild queueSkillsStatistics

    © 2026 mooncakes.io