moon-wfst

    Weighted finite-state transducer data model and graph algorithms in MoonBit

    wfst
    finite-state
    semiring
    transducer
    Download zip
    Version
    0.1.2
    License
    Apache-2.0
    Last updated
    16 days ago
    Downloads
    42

    No dependencies

    #MoonWFST

    MoonWFST 是可复用的 MoonBit 加权有限状态转导机(WFST)基础库。转导机用有限图表示输入与输出之间的关系:每条弧包含输入标签、输出标签及权重,权重由调用者提供的半环解释。本库面向语音候选格处理、形态分析、OCR 后处理、分词及其他有限状态处理流程,提供共享的数据模型和算法;它本身不是识别器,也不包含领域模型。

    版本与范围:0.1.2 修复图拓扑和路径等内部数组可被外部直接修改的问题,并增加边界回归。已实现能力是有明确边界的 WFST 子集,不是 OpenFst 的完整替代。下文示例使用源码中定义的合成图,不是生产数据或性能基准。

    #0.1.2 兼容性说明

    构图器与图快照的拓扑数组、接受路径轨迹、修剪映射均改为私有字段。请通过 transducer_arcs_from、transducer_final_weight、accepting_path_states、accepting_path_steps 和 trim_result_state_mapping 等公开访问器读取。公开函数签名不变;直接读取这些内部数组字段的代码需要迁移。可变权重载荷仍不提供深复制保证。旧版 0.1.1 不包含此修复。

    #已实现能力

    • 验证构图:密集状态编号、构建时复制的图快照、显式终止权重,以及与空字符串符号不同的 epsilon 标签。
    • 权重代数:调用者自定义半环,内置 Boolean 半环和有界非负 tropical 权重。
    • 关系变换:输入/输出投影、反转、带权并与连接,以及无 epsilon 的转导机组合。
    • 图分析:结构可达性、到终止状态的可达性、接受核心修剪、确定性拓扑顺序、入弧查询和图统计。
    • 路径查询:输入/输出 token 对的支持图成员判断、最低成本接受路径,以及 DAG 上有界的 k-best 路径。
    • 路径聚合:DAG 上通用半环的前向权重、后向权重和接受路径总权重。
    • 明确错误:非法状态、暂不支持的 epsilon 组合、Dijkstra 负权重、DAG 算法遇到环及非法路径数量限制。

    #安装与使用

    在已有 MoonBit 模块中安装精确版本:

    moon add okMambaOut/moon-wfst@0.1.2

    在使用该库的包目录中,通过 moon.pkg 指定导入别名:

    import {
    "okMambaOut/moon-wfst" @wfst,
    }

    通过 @wfst.transducer_builder 创建构图器,再使用 add_state、set_start、set_final、add_arc 和 build_transducer 构造关系图。完整的构图、组合、路径排序和结果提取流程见 语音候选格示例。

    模块名称为 okMambaOut/moon-wfst。版本与公开材料见 GitHub Release 和 MoonCakes 模块页面。

    #三个可运行示例

    安装 MoonBit 工具链后,在仓库根目录执行:

    moon run --target wasm-gc examples/speech_lattice moon run --target wasm-gc examples/morphology moon run --target wasm-gc examples/ocr_constraints

    示例输入与处理流程合成样例的确定结果
    语音候选格将识别候选格与词法映射组合,选择最低成本接受路径。cat,总成本 6。
    形态分析为不规则词形 went 的多个词素分析排序,提取低成本结果。go+PAST,成本 1。
    OCR 格式约束将六格字符候选与格式转导机组合,结合字形混淆成本重新评分。NB0158,总成本 6。

    这三个程序均位于 examples,可直接运行。OCR 示例不处理像素、不调用 OCR 模型、不验证校验码,结果不能解释为真实识别准确率;语音示例也不包含声学推理。

    #API 契约与功能边界

    • **半环:**调用者提供零元、单位元、候选相加及路径扩展运算。本库不能证明代数规律,也不能自动证明两个半环兼容;并与连接的合成弧使用左操作数的运算。
    • 转导机组合:两侧必须无 epsilon,并使用兼容且乘法可交换的半环。当前逐弧配对的权重交错不能视作一般非交换半环上的组合。任一侧出现 epsilon 均返回 UnsupportedEpsilonComposition,不隐含 epsilon 过滤器。
    • **最短路径:**Dijkstra 检查全部弧权重和终止权重均非负,包括不可达部分。调用者负责保证比较的单调性和代数一致性。
    • k-best:shortest_paths_in_dag 要求整张图无环。对于满足路径排序契约的 DAG,可使用适当的负边权;平行弧保留为不同推导,算法保存显式路径前缀,适用于有界 k 和中等规模图。
    • 成员判断:accepts_pair 根据调用者提供的零权重判定函数,回答支持图上的成员存在性。它不聚合权重、不返回路径见证,也不等价于“聚合总权重非零”;抵消、零因子或饱和运算可能使二者不同。内存规模与状态数 × 输入位置数 × 输出位置数成正比。
    • 前向/后向聚合:forward_weights_in_dag、backward_weights_in_dag 和 total_weight_in_dag 使用图的半环聚合路径权重,拒绝图中任何位置的环,不推断无限级数闭包。结果依赖调用者提供有效的半环运算。
    • **明确不包含:**带 epsilon 过滤的组合、带权 epsilon 闭包/消除、确定化、最小化、OpenFst 二进制导入导出、模型训练、分词策略、图像识别及领域专用校验。

    详细的运算顺序、结构行为与权重行为的区别、错误语义见 语义说明。有限检索范围内的选题比较与相邻项目边界见 查重记录。

    #构建与验证

    moon info moon fmt --check moon check --target wasm-gc --deny-warn moon check --target wasm --deny-warn moon check --target js --deny-warn moon check --target native --deny-warn moon build --target wasm-gc --deny-warn moon test --target wasm-gc --deny-warn

    2026-10-20,在本机 Moon 0.1.20261016 下,wasm-gc、WebAssembly 和 JavaScript 各通过 56/56 项仓库测试,各完成严格检查与构建。另在仓库外独立消费者中,以 Python 穷举路径、整数矩阵计算和循环图有限不动点计算作为预期,每个后端通过 208 项独立验算;这些测试不是性能或真实识别准确率结论。三个示例在三个本地后端均得到上述结果。

    本机 native 严格检查通过;构建实际尝试因缺少 MSVC C++ 工具和 Windows SDK 而失败,因此未运行本机 native 测试。GitHub Actions 工作流对每个发布提交分别执行四目标检查、构建、测试,以及三个 wasm-gc 示例;发布以对应 SHA 的真实运行结果为准,不沿用旧版 CI 证明新版。

    CI 对官方 Linux 2026-10-16 工具链及 core 下载执行固定 SHA-256 校验并核对版本;若官方 latest 内容变化会明确失败,维护者需重新验证后更新校验和与生成接口,不跳过门禁。安装脚本位于 .github/scripts/install-moonbit.sh。

    根目录基础库包含 1604 物理行非测试 .mbt 实现,计数包含空行,不包含测试、示例、生成接口或构建输出。这是实现规模与验证事实,不是性能结论。

    #许可证、来源与 AI 使用

    本项目采用 Apache-2.0,见 LICENSE 和 NOTICE。唯一导入的 MoonBit 包是工具链提供的 moonbitlang/core/debug,未随项目 vendoring。

    OpenFst 和 RustFST 仅用于术语、语义和 API 比较,未引入上游源码或数据。参考来源和范围见 第三方归属说明,AI 辅助使用情况见 AI-USE.md。

    #安全与资源限制

    库的输入是内存图,不是网络传输格式;库内没有网络访问、文件系统访问、动态加载或自动解析。处理不可信输入时,调用者必须限制状态数、弧数、token 长度和请求的 k。

    组合的状态规模可能达到两侧状态集的乘积,当前使用线性搜索查找状态对;Dijkstra 通过线性扫描选择下一状态,未使用堆;成员判断分配密集的状态/位置乘积空间;k-best 会复制路径前缀。这些算法不提供服务级的有界耗时保证,详见 资源限制。

    #项目历史与作者归属

    本项目在原有 MoonWFST 成果上继续开发,保留已有历史。最初 23 个提交保留原来的 yujieMamba 作者归属,后续接手提交使用参赛者指定的仓库级名称 okmanba 和 okMambaOut GitHub noreply 邮箱。

    仓库所有权与提交作者是不同事实;本项目未改写历史,也不声称两个账号代表同一自然人。

    AcceptingPath

    pub struct AcceptingPath[W] {
    final_weight : W
    total : W
    // private fields
    }

    A least-cost accepting path with its terminal weight included in total.

    Arc

    pub(all) struct Arc[W] {
    input : Label
    output : Label
    destination : Int
    weight : W
    }

    One directed, weighted transition. Its source is the containing state's id.

    Label

    pub(all) enum Label {
    Epsilon
    Symbol(String)
    } derive(Eq,
    Debug
    )

    An explicit transducer label. Epsilon is not represented by an empty string.

    Label::equal

    fn Label::equal(Label, Label) -> Bool

    Label::not_equal

    fn Label::not_equal(x : Label, y : Label) -> Bool

    Label::to_repr

    PathOrder

    pub struct PathOrder[W] {
    compare : (W, W) -> Int
    is_nonnegative : (W) -> Bool
    is_zero : (W) -> Bool
    }

    Ordering and validation operations required by Dijkstra's algorithm. The caller must ensure comparison agrees with semiring alternatives and that extending a path by a non-negative weight is monotone.

    PathStep

    pub(all) struct PathStep[W] {
    source : Int
    arc : Arc[W]
    }

    One traversed arc annotated with the source state.

    Semiring

    pub(all) struct Semiring[W] {
    zero : W
    one : W
    plus : (W, W) -> W
    times : (W, W) -> W
    }

    Semiring operations used by graph construction and algorithms. The caller is responsible for satisfying the semiring laws.

    Transducer

    pub struct Transducer[W] {
    semiring : Semiring[W]
    start : Int?
    // private fields
    }

    Validated weighted finite-state transducer. Graph arrays are not exposed as mutable references; use the accessors to inspect a snapshot.

    TransducerBuilder

    pub struct TransducerBuilder[W] {
    semiring : Semiring[W]
    start : Int?
    // private fields
    }

    Mutable graph assembler. A builder can be reused after build; every built graph owns copies of its adjacency and final-weight arrays.

    TransducerStats

    pub(all) struct TransducerStats {
    state_count : Int
    arc_count : Int
    epsilon_input_arc_count : Int
    epsilon_output_arc_count : Int
    final_state_count : Int
    max_out_degree : Int
    reachable_state_count : Int
    coaccessible_state_count : Int
    has_start : Bool
    } derive(Eq)

    Structural metrics captured from a transducer snapshot. Epsilon counts are independent: an epsilon/epsilon arc contributes to both counters.

    TransducerStats::equal

    TransducerStats::not_equal

    fn TransducerStats::not_equal(x : TransducerStats, y : TransducerStats) -> Bool

    TrimResult

    pub struct TrimResult[W] {
    graph : Transducer[W]
    // private fields
    }

    Compact graph together with a mapping from each original state to its new identifier. None means that the original state was removed.

    TropicalWeight

    pub enum TropicalWeight {
    Finite(Int)
    Infinity
    } derive(
    Debug
    )

    A bounded tropical weight. Finite values are non-negative machine integers; Infinity is the additive identity and represents no path.

    WfstError

    pub(all) enum WfstError {
    InvalidStateId(Int)
    NoStartState
    UnsupportedEpsilonComposition
    GraphHasCycle
    InvalidPathLimit(Int)
    NegativeWeight
    WeightOutOfRange
    NoAcceptingPath
    } derive(Eq,
    Debug
    )

    Stable errors returned by graph and algorithm APIs.

    WfstError::equal

    fn WfstError::equal(WfstError, WfstError) -> Bool

    WfstError::not_equal

    fn WfstError::not_equal(x : WfstError, y : WfstError) -> Bool

    accepting_core_states

    fn[W] accepting_core_states(transducer : Transducer[W]) -> Array[Bool]

    Return the states lying on at least one structural path from the declared start to an explicitly final state. Weights and labels are not interpreted.

    accepting_path_final_weight

    fn[W] accepting_path_final_weight(path : AcceptingPath[W]) -> W

    The final-state weight attached to the accepting state.

    accepting_path_states

    fn[W] accepting_path_states(path : AcceptingPath[W]) -> Array[Int]

    Return a copy of the state sequence from start through accepting state.

    accepting_path_steps

    fn[W] accepting_path_steps(path : AcceptingPath[W]) -> Array[PathStep[W]]

    Return a copy of the arc trace.

    accepting_path_total_weight

    fn[W] accepting_path_total_weight(path : AcceptingPath[W]) -> W

    The product of all arc weights followed by the final-state weight.

    accepts_pair

    fn[W] accepts_pair(transducer : Transducer[W], input : Array[String], output : Array[String], is_zero : (W) -> Bool) -> Result[Bool, WfstError]

    Check whether a transducer has a path matching both complete token arrays. Epsilon labels consume no token on their respective side. is_zero lets the caller exclude semiring-zero arcs and finals; all other weight operations are ignored. This is a membership query, not a path-weight aggregation API.

    add_arc

    fn[W] add_arc(builder : TransducerBuilder[W], source : Int, input : Label, output : Label, destination : Int, weight : W) -> Result[Unit, WfstError]

    Add an arc after validating both endpoint identifiers.

    add_state

    fn[W] add_state(builder : TransducerBuilder[W]) -> Int

    Append a state and return its dense, zero-based identifier.

    backward_weights_in_dag

    fn[W] backward_weights_in_dag(transducer : Transducer[W]) -> Result[Array[W], WfstError]

    Aggregate semiring weights of every accepting suffix from each state in a DAG. Explicit final weights contribute paths that stop at that state; arcs may still continue to other accepting paths. States with no accepting suffix have semiring zero. The entire graph must be acyclic.

    boolean_semiring

    fn boolean_semiring() -> Semiring[Bool]

    Boolean weights: false is zero, true is one, alternatives use OR, and path extension uses AND.

    build_transducer

    fn[W] build_transducer(builder : TransducerBuilder[W]) -> Transducer[W]

    Produce a graph snapshot independent of later builder mutations.

    clear_final

    fn[W] clear_final(builder : TransducerBuilder[W], state : Int) -> Result[Unit, WfstError]

    Mark a state as non-final.

    coaccessible_states

    fn[W] coaccessible_states(transducer : Transducer[W]) -> Array[Bool]

    Return whether each state can reach a structurally final state. A state is final whenever it has an explicitly stored final weight, even if that value is the semiring zero. Epsilon and zero-valued arcs are traversed.

    compose

    fn[W] compose(left : Transducer[W], right : Transducer[W]) -> Result[Transducer[W], WfstError]

    Compose two epsilon-free transducers. A left output label is joined with an equal right input label. Both operands must use compatible semiring laws, and multiplication must be commutative: arc-wise interleaving otherwise differs from multiplying the complete left path by the complete right path. These algebraic obligations cannot be checked at runtime. The result uses the left graph's semiring.

    concatenate_transducers

    fn[W] concatenate_transducers(left : Transducer[W], right : Transducer[W]) -> Transducer[W]

    Concatenate two weighted transducers. Each final weight in the left graph becomes an epsilon transition to the right start; its value is carried on that transition so path multiplication order remains left-to-right. Both inputs must use semiring operations with compatible meanings.

    forward_weights_in_dag

    fn[W] forward_weights_in_dag(transducer : Transducer[W]) -> Result[Array[W], WfstError]

    Aggregate semiring weights of every prefix reaching each state in a DAG. The result is indexed by dense state identifier; states unreachable from the declared start retain semiring zero. The entire graph must be acyclic, including components that are not reachable from the start.

    invert

    fn[W] invert(transducer : Transducer[W]) -> Transducer[W]

    Reverse every arc's input/output mapping while preserving graph topology and all weights.

    label_is_epsilon

    fn label_is_epsilon(label : Label) -> Bool

    Return whether a label is the explicit epsilon marker.

    path_order

    fn[W] path_order(compare : (W, W) -> Int, is_nonnegative : (W) -> Bool, is_zero : (W) -> Bool) -> PathOrder[W]

    Construct a path-order policy for a caller-defined weight domain.

    project_input

    fn[W] project_input(transducer : Transducer[W]) -> Transducer[W]

    Copy a transducer while replacing each output label with its input label.

    project_output

    fn[W] project_output(transducer : Transducer[W]) -> Transducer[W]

    Copy a transducer while replacing each input label with its output label.

    reachable_states

    fn[W] reachable_states(transducer : Transducer[W]) -> Array[Bool]

    Return whether each state can be reached from the start by graph arcs. Arc weights and labels (including epsilon) do not affect structural reachability.

    reachable_states_from

    fn[W] reachable_states_from(transducer : Transducer[W], root : Int) -> Result[Array[Bool], WfstError]

    Compute structural reachability from an explicit root state rather than the graph's declared start. This is useful when inspecting components or when an algorithm chooses its own entry state. Every stored arc is traversed.

    semiring_one

    fn[W] semiring_one(semiring : Semiring[W]) -> W

    The multiplicative identity of a semiring.

    semiring_plus

    fn[W] semiring_plus(semiring : Semiring[W], left : W, right : W) -> W

    Apply semiring addition to two alternative path weights.

    semiring_times

    fn[W] semiring_times(semiring : Semiring[W], left : W, right : W) -> W

    Extend a path by composing its current weight with one arc weight.

    semiring_zero

    fn[W] semiring_zero(semiring : Semiring[W]) -> W

    The additive identity of a semiring.

    set_final

    fn[W] set_final(builder : TransducerBuilder[W], state : Int, weight : W) -> Result[Unit, WfstError]

    Set a state's final weight. Passing a semiring-zero weight is permitted and retains the distinction between a final state with zero weight and non-final.

    set_start

    fn[W] set_start(builder : TransducerBuilder[W], state : Int) -> Result[Unit, WfstError]

    Set the start state. An invalid state id leaves the builder unchanged.

    shortest_path

    fn[W] shortest_path(transducer : Transducer[W], order : PathOrder[W]) -> Result[AcceptingPath[W], WfstError]

    Find a least-cost accepting path using an O(V^2 + E) linear-scan Dijkstra implementation. Every arc and final weight is validated, including values on unreachable states. Equal-cost alternatives preserve state-id and arc insertion order. The supplied PathOrder laws cannot be proven at runtime.

    shortest_paths_in_dag

    fn[W] shortest_paths_in_dag(transducer : Transducer[W], order : PathOrder[W], limit : Int) -> Result[Array[AcceptingPath[W]], WfstError]

    Find up to limit least-weight accepting paths in a finite DAG. Unlike shortest_path, this dynamic program does not require nonnegative arc weights because cycles are rejected before traversal. The supplied comparison must still be compatible with semiring path extension. Parallel arcs remain distinct paths; equal-cost paths retain deterministic graph traversal order. A nonpositive limit is rejected.

    topological_order

    fn[W] topological_order(transducer : Transducer[W]) -> Result[Array[Int], WfstError]

    Return a deterministic topological order of every state in a DAG. All disconnected components are included. Initially ready states are queued in ascending state-id order; newly ready states follow outgoing-arc insertion order and are appended to the queue. A cycle returns GraphHasCycle.

    total_weight_in_dag

    fn[W] total_weight_in_dag(transducer : Transducer[W]) -> Result[W, WfstError]

    Return the semiring aggregate of all accepting paths in a DAG. A graph without a declared start denotes no paths and returns semiring zero. This operation rejects cycles rather than assuming a finite closure exists.

    transducer_arcs_from

    fn[W] transducer_arcs_from(transducer : Transducer[W], state : Int) -> Result[Array[Arc[W]], WfstError]

    Return a copy of a state's outgoing arcs. Invalid identifiers are reported.

    transducer_builder

    fn[W] transducer_builder(semiring : Semiring[W]) -> TransducerBuilder[W]

    Create an empty transducer builder for the supplied semiring.

    transducer_final_weight

    fn[W] transducer_final_weight(transducer : Transducer[W], state : Int) -> Result[W?, WfstError]

    Read a state's optional final weight.

    transducer_has_epsilon

    fn[W] transducer_has_epsilon(transducer : Transducer[W]) -> Bool

    Return whether any arc uses epsilon on either label side.

    transducer_incoming_arcs

    fn[W] transducer_incoming_arcs(transducer : Transducer[W], destination : Int) -> Result[Array[(Int, Arc[W])], WfstError]

    Return a stable copy of all arcs entering destination, ordered first by source-state id and then by each source's insertion order. Parallel arcs are retained as separate entries. The returned array is independent of the graph.

    transducer_is_trimmed

    fn[W] transducer_is_trimmed(transducer : Transducer[W]) -> Bool

    Report whether every state belongs to the graph's structural accepting core. The empty graph is trimmed; a graph without a start and with states is not. This check ignores labels and semiring-zero values.

    transducer_semiring

    fn[W] transducer_semiring(transducer : Transducer[W]) -> Semiring[W]

    Return the semiring attached to this graph.

    transducer_start_state

    fn[W] transducer_start_state(transducer : Transducer[W]) -> Int?

    Read the optional start-state identifier.

    transducer_state_count

    fn[W] transducer_state_count(transducer : Transducer[W]) -> Int

    Number of states in a transducer snapshot.

    transducer_stats

    fn[W] transducer_stats(transducer : Transducer[W]) -> TransducerStats

    Count graph structure and start/final reachability in O(V + E) time. Counts include unreachable states and explicitly stored zero-valued finals.

    trim

    fn[W] trim(transducer : Transducer[W]) -> TrimResult[W]

    Remove states that are not both reachable from the start and able to reach a structurally final state. Retained states are re-numbered in original state order. An absent start or an empty accepting core yields an empty graph with no start state. The operation is structural: it does not inspect label matches or compare generic weights against semiring zero.

    trim_result_state_mapping

    fn[W] trim_result_state_mapping(result : TrimResult[W]) -> Array[Int?]

    Return a copy of the original-to-compacted state mapping.

    trim_result_transducer

    fn[W] trim_result_transducer(result : TrimResult[W]) -> Transducer[W]

    Read the compacted graph produced by trim.

    tropical_compare

    fn tropical_compare(left : TropicalWeight, right : TropicalWeight) -> Int

    Compare tropical weights using their natural least-cost order.

    tropical_cost

    fn tropical_cost(cost : Int) -> Result[TropicalWeight, WfstError]

    Construct a finite tropical cost. Negative and reserved infinity values are rejected so every public finite value has well-defined saturating addition.

    tropical_infinity

    fn tropical_infinity() -> TropicalWeight

    Return the tropical additive identity.

    tropical_is_infinite

    fn tropical_is_infinite(weight : TropicalWeight) -> Bool

    Return whether a tropical weight is infinite.

    tropical_path_order

    fn tropical_path_order() -> PathOrder[TropicalWeight]

    Dijkstra policy for the built-in bounded tropical semiring.

    tropical_semiring

    fn tropical_semiring() -> Semiring[TropicalWeight]

    A bounded non-negative min-plus semiring suitable for deterministic least-cost graph examples. Addition saturates to infinity before overflow.

    tropical_value

    fn tropical_value(weight : TropicalWeight) -> Int?

    Return a finite cost, or None for tropical infinity.

    union_transducers

    fn[W] union_transducers(left : Transducer[W], right : Transducer[W]) -> Transducer[W]

    Form the weighted union of two transducers using a fresh epsilon branching state. Both inputs must use semiring operations with compatible meanings.