🥮 mooncakes.io

moonalign

A MoonBit-native bilingual paragraph and sentence alignment toolkit with dynamic-programming based matching and a practical CLI.

alignment
bilingual
nlp
translation
corpus
moon add LL728/moonalign@0.2.1
Download zip
Author
Version
0.2.1
License
Apache-2.0
Repository
Last updated
3 hours ago
Downloads
7
README

#MoonAlign

MoonAlign 是一个使用 MoonBit 编写的双语文本对齐工具包，用于把段落或句子级的源文本与目标文本转换为可审查、可评估、可继续处理的对齐结果。

它适合用于平行语料构建、翻译流程清洗、术语对照、双语文档质量检查，以及需要轻量、可解释对齐能力的文本工程工具。项目提供 MoonBit 库 API 和命令行界面，不依赖在线服务或大型模型即可运行。

#项目定位

MoonAlign 关注“稳定的对齐底座”，而不是翻译或语义质量的黑盒替代品。它使用透明的长度特征、词元特征和动态规划搜索，在保持可解释性的同时处理常见的 1-1、1-2、2-1 对齐关系。

核心设计目标：

  • MoonBit 原生实现，便于嵌入其他 MoonBit 项目。
  • 离线运行，结果稳定，可在 CI 和数据流水线中复现。
  • 输出结构化报告，保留跨度、得分、比例、告警和质量指标。
  • 通过词法锚点、质量门禁和审校队列支持人工复核。

#核心能力

  • 句子级和段落级切分，兼容中英文标点、Unicode 空白和 CRLF 文本。
  • 基于动态规划的有限扇出对齐，支持 1-1、1-2、2-1 等合并路径。
  • ASCII 词元、CJK 字符和标点的轻量加权长度模型。
  • URL、数字、标识符和共享词元的词法锚点提取。
  • 单文档和批量语料 API，支持输入校验、批量对齐和结果汇总。
  • 质量评估：覆盖率、置信度、单调性、合并比例、告警和质量门禁。
  • 人工审校队列，以及 JSON、TSV、CSV 和特征向量导出。
  • 210 维对齐特征和 210 维文本单元特征，可用于排序和下游校准实验。

#快速开始

#环境要求

  • MoonBit CLI stable
  • Git

在项目根目录运行：

moon check
moon test

#库 API

///|
test "basic alignment" {
  let report = @moonalign.align(
    "MoonBit favors maintainable tooling.\n", "MoonBit 强调可维护的工具链。\n",
  )
  inspect(report.pairs.length(), content="1")
  inspect(report.pairs[0].move_kind, content="1-1")
}

批量处理和质量审校：

///|
let documents = @moonalign.synthetic_corpus(10)

///|
let results = @moonalign.align_corpus(documents)

///|
let summary = @moonalign.summarize_corpus(results)

///|
let review = @moonalign.make_review_queue(results[0])

#安装为依赖

项目模块名为 LL728/moonalign，版本和仓库信息位于 moon.mod。在其他 MoonBit 项目中添加依赖后，可通过 @moonalign 使用公开 API。

#CLI

MoonAlign CLI 位于 cmd/main，支持直接传入文本并输出 JSON 或 TSV：

moon run cmd/main -- \
  --source-text "MoonBit favors stable tooling. MoonAlign builds corpora." \
  --target-text "MoonBit 强调稳定工具链。MoonAlign 用来构建语料。" \
  --mode sentence \
  --format json

常用参数：

参数说明
--source-text <text>源语言文本
--target-text <text>目标语言文本
--mode sentence\|paragraph句子级或段落级切分，默认 sentence
--format json\|tsvJSON 或 TSV 输出，默认 json
--benchmark运行内置离线基准并输出 JSON 指标
--help显示帮助

输出结果包含源/目标跨度、移动类型、文本内容、长度权重、逐步得分和诊断告警。

运行基准：

moon run --target wasm-gc cmd/main -- --benchmark

#架构

source / target text
        │
        ▼
normalization and segmentation
        │
        ▼
weighted text units ─────── lexical anchors
        │                            │
        └──────── dynamic programming alignment
                                      │
                                      ▼
                         alignment report and metrics
                                      │
                   ┌──────────────────┼──────────────────┐
                   ▼                  ▼                  ▼
                 JSON/TSV          quality gate       review queue

主要模块：

  • normalize.mbt：空白归一化、段落识别、句子切分和文本权重计算。
  • aligner.mbt：有限扇出动态规划、路径回溯和对齐跨度生成。
  • anchors.mbt：共享词元、URL、数字和标识符锚点。
  • metrics.mbt、quality.mbt：gold set 评估和质量门禁。
  • corpus.mbt、review.mbt：批量处理、导出和人工审校工作流。
  • feature_engineering.mbt：用于排序和校准的确定性特征向量。
  • cmd/main/main.mbt：命令行参数解析和输出格式选择。

#基准

仓库包含可离线运行的真实小样本基准，数据文件位于 benchmarks/tatoeba-eng-cmn.tsv，并保留 Tatoeba sentence ID、来源链接和许可证说明。

当前基准输出包括：

数据集PrecisionRecallF1覆盖率
tatoeba-eng-cmn-short0.25000.22220.23531.0000
tatoeba-eng-cmn-merge1.00001.00001.00001.0000

第一个样本专门暴露长度启发式在短句和中英文长度差异下的边界；第二个样本用于保证 1-2 合并路径持续可用。基准不是翻译质量分数，也不宣称代表大规模语料分布。

更多来源和数据许可信息见 benchmarks/README.md 与 SOURCES.md。

#测试

测试覆盖基础对齐、切分、归一化、gold set 指标、批量 API、质量门禁、特征向量、CLI smoke test 和边界输入。

本地推荐执行：

moon fmt --check
moon check --target all --deny-warn
moon test --target wasm-gc --deny-warn
moon test --target native --deny-warn
moon info

边界回归测试覆盖空文本、空白、CRLF、Unicode、中文/英文混合、数字、URL、标点、段落空行、极端长度比和合并边界。

#CI

GitHub Actions 工作流位于 .github/workflows/check.yml，在 ubuntu-latest、macos-latest 和 windows-latest 上运行，并执行：

  • stable MoonBit CLI 安装和工具链同步；
  • 格式检查、全后端类型检查和 warning 拒绝；
  • wasm-gc 与 native 测试；
  • 公共接口生成和 git diff 漂移检查；
  • 离线基准回归和 CLI smoke test；
  • 边界测试数量检查。

#许可证

MoonAlign 使用 Apache License 2.0 发布。仓库中的 Tatoeba 基准数据遵循其来源许可证并保留必要署名；使用或再分发该数据时，请同时阅读 SOURCES.md 和 benchmarks/README.md。

#相关文档

#
AlignOptions

source
pub struct AlignOptions {
  segment_mode : SegmentMode
  max_fan_out : Int
  join_penalty : Double
  deviation_penalty : Double
  min_sentence_chars : Int
  preserve_paragraphs : Bool
} derive(
Debug)

#
AlignmentMetrics

source
pub struct AlignmentMetrics {
  predicted_pairs : Int
  gold_pairs : Int
  exact_pairs : Int
  precision : Double
  recall : Double
  f1 : Double
  source_coverage : Double
  target_coverage : Double
  merged_pairs : Int
  average_score : Double
} derive(ToJson, 
Debug)

#
AlignmentPair

source
pub struct AlignmentPair {
  source_start : Int
  source_end : Int
  target_start : Int
  target_end : Int
  source_text : String
  target_text : String
  source_char_weight : Double
  target_char_weight : Double
  source_tokens : Int
  target_tokens : Int
  score : Double
  move_kind : String
} derive(ToJson, 
Debug)

#
AlignmentQuality

source
pub struct AlignmentQuality {
  source_units : Int
  target_units : Int
  aligned_pairs : Int
  exact_one_to_one : Int
  merged_pairs : Int
  anchored_pairs : Int
  source_coverage : Double
  target_coverage : Double
  mean_confidence : Double
  monotonicity : Double
  score : Double
  issues : Array[QualityIssue]
} derive(ToJson, 
Debug)

Quality statistics suitable for CI regression checks.

#
AlignmentReport

source
pub struct AlignmentReport {
  options : Json
  source_count : Int
  target_count : Int
  estimated_ratio : Double
  warnings : Array[String]
  pairs : Array[AlignmentPair]
} derive(ToJson, 
Debug)

#
AnchorOptions

source
pub struct AnchorOptions {
  min_token_length : Int
  include_numbers : Bool
  include_urls : Bool
  include_identifiers : Bool
  case_sensitive : Bool
  max_anchors_per_unit : Int
} derive(
Debug)

A configurable set of deterministic anchor extraction rules.

#
BackPointer

source
type BackPointer derive(Eq, 
Debug)

#
BenchmarkCase

source
pub struct BenchmarkCase {
  name : String
  source : String
  target : String
  gold_pairs : Array[GoldPair]
  source_url : String
  license : String
} derive(ToJson, 
Debug)

#
BenchmarkResult

source
pub struct BenchmarkResult {
  name : String
  metrics : AlignmentMetrics
  warnings : Array[String]
} derive(ToJson, 
Debug)

#
CorpusDocument

source
pub struct CorpusDocument {
  id : String
  source : String
  target : String
  metadata : Map[String, String]
} derive(
Debug)

A named bilingual document in a batch.

#
CorpusDocument::with_metadata

source
fn CorpusDocument::with_metadata(document : CorpusDocument, key~ : String, value~ : String) -> CorpusDocument

Add or replace a metadata field on a corpus document.

#
CorpusResult

source
pub struct CorpusResult {
  id : String
  report : AlignmentReport
  quality : AlignmentQuality
} derive(ToJson, 
Debug)

A batch alignment result with aggregate quality information.

#
CorpusSummary

source
pub struct CorpusSummary {
  document_count : Int
  total_source_units : Int
  total_target_units : Int
  total_pairs : Int
  mean_f1 : Double
  mean_quality_score : Double
  warning_count : Int
  issue_count : Int
} derive(ToJson, 
Debug)

Aggregate statistics over a batch run.

#
GoldPair

source
pub struct GoldPair {
  source_start : Int
  source_end : Int
  target_start : Int
  target_end : Int
} derive(Eq, ToJson, 
Debug)

#
LengthStats

source
pub struct LengthStats {
  count : Int
  total : Double
  minimum : Double
  maximum : Double
  mean : Double
  median : Double
  p90 : Double
} derive(ToJson, 
Debug)

A transparent distribution of weighted text lengths.

#
LexicalAnchor

source
pub struct LexicalAnchor {
  source_unit : Int
  target_unit : Int
  source_text : String
  target_text : String
  normalized : String
  kind : String
  score : Double
} derive(ToJson, 
Debug)

A lexical anchor found in both sides of a bilingual document.

Anchors are intentionally lightweight: they are useful for terminology, identifiers, URLs, numbers, and names without requiring a language model.

#
QualityGate

source
pub struct QualityGate {
  min_source_coverage : Double
  min_target_coverage : Double
  min_mean_confidence : Double
  min_quality_score : Double
  max_warning_count : Int
} derive(
Debug)

Thresholds used by automated quality gates.

#
QualityIssue

source
pub struct QualityIssue {
  code : String
  severity : String
  message : String
  source_unit : Int?
  target_unit : Int?
} derive(ToJson, 
Debug)

A document-level quality diagnostic.

#
ReviewItem

source
pub struct ReviewItem {
  document_id : String
  pair_index : Int
  label : ReviewLabel
  confidence : Double
  reason : String
  anchors : Int
} derive(ToJson, 
Debug)

A pair plus the evidence used to make a review decision.

#
ReviewLabel

source
pub enum ReviewLabel {
  Accept
  NeedsReview
  Reject
} derive(Eq, ToJson, 
Debug)

A review label assigned to an alignment pair.

#
ReviewQueue

source
pub struct ReviewQueue {
  items : Array[ReviewItem]
  accepted : Int
  needs_review : Int
  rejected : Int
} derive(ToJson, 
Debug)

An ordered human-review queue.

#
SegmentMode

source
pub enum SegmentMode {
  Paragraph
  Sentence
} derive(Eq, 
Debug)

#
TextUnit

source
pub struct TextUnit {
  id : Int
  paragraph_index : Int
  sentence_index : Int
  text : String
  normalized : String
  char_weight : Double
  token_count : Int
} derive(ToJson, 
Debug)

#
accepted_documents

source
fn accepted_documents(results : Array[CorpusResult], gate? : QualityGate) -> Array[CorpusResult]

Filter batch results to documents that pass a quality gate.

#
align

source
fn align(source_text : String, target_text : String, options? : AlignOptions) -> AlignmentReport

#
align_corpus

source
fn align_corpus(documents : Array[CorpusDocument], options? : AlignOptions) -> Array[CorpusResult]

Align every document in a batch while preserving input order.

#
align_document

source
fn align_document(document : CorpusDocument, options? : AlignOptions) -> CorpusResult

#
align_texts

source
fn align_texts(source_text : String, target_text : String, options? : AlignOptions) -> AlignmentReport

#
align_units

source
fn align_units(source_units : Array[TextUnit], target_units : Array[TextUnit], options? : AlignOptions) -> AlignmentReport

#
align_with_mode

source
fn align_with_mode(source_text : String, target_text : String, mode? : SegmentMode) -> AlignmentReport

#
alignment_feature_mean

source
fn alignment_feature_mean(pair : AlignmentPair) -> Double

Return a compact aggregate of an alignment feature vector.

#
alignment_feature_vector

source
fn alignment_feature_vector(pair : AlignmentPair) -> Array[Double]

Extract the alignment feature vector for ranking and calibration.

#
anchor_bonus

source
fn anchor_bonus(pair : AlignmentPair, anchors : Array[LexicalAnchor]) -> Double

Compute an anchor bonus for a proposed alignment pair.

#
anchor_kind_counts

source
fn anchor_kind_counts(anchors : Array[LexicalAnchor]) -> Map[String, Int]

Count anchors grouped by their category.

#
assess_quality

source
fn assess_quality(report : AlignmentReport, anchors? : Array[LexicalAnchor]) -> AlignmentQuality

Build a quality report from alignment output and lexical evidence.

#
benchmark_case

source
fn benchmark_case(case : BenchmarkCase) -> BenchmarkResult

#
benchmark_cases

source
fn benchmark_cases() -> Array[BenchmarkCase]

#
benchmark_suite

source
fn benchmark_suite() -> Array[BenchmarkResult]

#
benchmark_to_json_string

source
fn benchmark_to_json_string(indent? : Int) -> String

#
corpus_document

source
fn corpus_document(id~ : String, source~ : String, target~ : String) -> CorpusDocument

Create a corpus document without requiring a mutable metadata map.

#
corpus_quality_csv

source
fn corpus_quality_csv(results : Array[CorpusResult]) -> String

Render a compact quality dashboard as CSV.

#
corpus_to_tsv

source
fn corpus_to_tsv(results : Array[CorpusResult]) -> String

Render aligned corpus pairs in a review-friendly TSV format.

#
default_anchor_options

source
fn default_anchor_options() -> AnchorOptions

Return conservative defaults for lexical anchor extraction.

#
default_options

source
fn default_options() -> AlignOptions

#
default_quality_gate

source
fn default_quality_gate() -> QualityGate

Return thresholds appropriate for human review queues.

#
evaluate

source
fn evaluate(report : AlignmentReport, gold_pairs : Array[GoldPair]) -> AlignmentMetrics

#
extract_anchors

source
fn extract_anchors(units : Array[TextUnit], options? : AnchorOptions) -> Array[LexicalAnchor]

Extract deterministic anchors from segmented units.

#
feature_vector_to_csv

source
fn feature_vector_to_csv(values : Array[Double]) -> String

Convert a feature vector into a stable comma-separated row.

#
gold_pair

source
fn gold_pair(source_start~ : Int, source_end~ : Int, target_start~ : Int, target_end~ : Int) -> GoldPair

#
length_ratio

source
fn length_ratio(source : Array[TextUnit], target : Array[TextUnit]) -> Double

Compute a weighted length ratio for two segmented documents.

#
length_stats

source
fn length_stats(units : Array[TextUnit]) -> LengthStats

Compute min/max/mean/median/p90 from weighted units.

#
make_review_queue

source
fn make_review_queue(result : CorpusResult) -> ReviewQueue

Build an ordered review queue from one document result.

#
make_review_queues

source
fn make_review_queues(results : Array[CorpusResult]) -> Array[ReviewQueue]

Build queues for a complete batch.

#
match_anchors

source
fn match_anchors(source_units : Array[TextUnit], target_units : Array[TextUnit], options? : AnchorOptions) -> Array[LexicalAnchor]

Create cross-language anchor candidates using shared normalized tokens.

#
move_kind_counts

source
fn move_kind_counts(results : Array[CorpusResult]) -> Map[String, Int]

Count pairs by move shape for regression monitoring.

#
normalize_text

source
fn normalize_text(text : String) -> String

#
options_to_json

source
fn options_to_json(options : AlignOptions) -> Json

#
paragraph_histogram

source
fn paragraph_histogram(units : Array[TextUnit]) -> Map[Int, Int]

Count units by paragraph while keeping empty paragraphs visible in reports.

#
paragraph_mode

source
fn paragraph_mode() -> SegmentMode

#
parse_corpus_tsv

source
fn parse_corpus_tsv(text : String) -> Array[CorpusDocument]

Parse a minimal tab-separated corpus input. Blank and comment lines skip.

#
passes_quality_gate

source
fn passes_quality_gate(quality : AlignmentQuality, gate? : QualityGate) -> Bool

Check whether a quality result is acceptable for automated ingestion.

#
pending_review_items

source
fn pending_review_items(queue : ReviewQueue) -> Array[ReviewItem]

Return only items which should be shown to a human.

#
quality_issue

source
fn quality_issue(code~ : String, severity~ : String, message~ : String, source_unit? : Int, target_unit? : Int) -> QualityIssue

Create a quality issue with optional unit locations.

#
quality_recommendations

source
fn quality_recommendations(quality : AlignmentQuality) -> Array[String]

Produce human-readable review actions from quality diagnostics.

#
quality_to_csv_row

source
fn quality_to_csv_row(id : String, quality : AlignmentQuality) -> String

Return a stable CSV row for quality dashboards.

#
repeated_units

source
fn repeated_units(units : Array[TextUnit]) -> Map[String, Int]

Return all repeated normalized units and their frequency.

#
report_to_json_string

source
fn report_to_json_string(report : AlignmentReport, indent? : Int) -> String

#
report_to_tsv

source
fn report_to_tsv(report : AlignmentReport) -> String

#
review_label_counts

source
fn review_label_counts(queues : Array[ReviewQueue]) -> Map[String, Int]

Count review labels across a batch.

#
review_queue

source
fn review_queue(results : Array[CorpusResult]) -> Array[CorpusResult]

Filter batch results to documents requiring human review.

#
review_queue_to_tsv

source
fn review_queue_to_tsv(queue : ReviewQueue) -> String

Render a review queue as TSV for spreadsheet workflows.

#
review_summary

source
fn review_summary(queue : ReviewQueue) -> String

Return a stable summary sentence for a review dashboard.

#
segment_text

source
fn segment_text(text : String, options? : AlignOptions) -> Array[TextUnit]

#
sentence_mode

source
fn sentence_mode() -> SegmentMode

#
short_unit_indexes

source
fn short_unit_indexes(units : Array[TextUnit], threshold? : Double) -> Array[Int]

Return all unit indexes whose normalized text is unusually short.

#
stats_to_csv_row

source
fn stats_to_csv_row(label : String, stats : LengthStats) -> String

Serialize basic statistics as one stable CSV row.

#
summarize_corpus

source
fn summarize_corpus(results : Array[CorpusResult]) -> CorpusSummary

Summarize batch output for CI and release notes.

#
synthetic_corpus

source
fn synthetic_corpus(count : Int) -> Array[CorpusDocument]

Build a deterministic synthetic corpus used by performance tests.

#
text_feature_peak

source
fn text_feature_peak(unit : TextUnit) -> Double

Return the largest feature value for a text unit.

#
text_feature_vector

source
fn text_feature_vector(unit : TextUnit) -> Array[Double]

Extract the text-unit feature vector for segmentation experiments.

#
text_profile

source
fn text_profile(text : String, options? : AlignOptions) -> String

Produce a deterministic text profile used in issue reports.

#
token_overlap

source
fn token_overlap(left : TextUnit, right : TextUnit) -> Double

Return normalized token overlap between two units.

#
validate_corpus

source
fn validate_corpus(documents : Array[CorpusDocument]) -> Array[String]

Validate a batch before running expensive alignment.

#
with_max_fan_out

source
fn with_max_fan_out(options : AlignOptions, max_fan_out : Int) -> AlignOptions

#
with_min_sentence_chars

source
fn with_min_sentence_chars(options : AlignOptions, min_sentence_chars : Int) -> AlignOptions

#
with_preserve_paragraphs

source
fn with_preserve_paragraphs(options : AlignOptions, preserve : Bool) -> AlignOptions

#
with_segment_mode

source
fn with_segment_mode(options : AlignOptions, mode : SegmentMode) -> AlignOptions

Source Files

Powered by MoonBit

Site sourceReport issuePackagesBuild queueSkillsStatistics

© 2026 mooncakes.io