🥮 mooncakes.io

    moonarpa

    Portable ARPA backoff language models, corpus scoring, rescoring and count training

    arpa
    ngram
    language-model
    backoff
    perplexity
    Download zip
    Version
    0.1.0
    License
    MIT
    Repository
    Last updated
    22 hours ago
    Downloads
    12

    No dependencies

    #MoonARPA

    MoonARPA 是纯 MoonBit 的词级 n-gram 语言模型库，提供 ARPA 模型读写、条件回退概率、句子与语料评分、候选重排和小语料训练。模块为 TerryFFF-jenius/moonarpa@0.1.0，不依赖网络服务和外部模型运行时。

    #适用任务

    • 语音/OCR 工程：输入已经分词的候选句及声学分数，用语言模型重新排序，保留稳定的候选 ID。
    • 语料工程：固定训练词表，将留出集映射后计算逐词似然、OOV、交叉熵和语料困惑度。
    • 离线应用：合并语料分片计数，训练小型模型，导出 ARPA 后恢复，并查询下一个词或维护流式评分状态。

    这些工作流可通过库 API 或 JSON 任务协议接入多个项目。它不提供分词器、神经推理、通用 WFST、搜索索引、KenLM 二进制文件兼容或大规模并行训练。

    #环境与安装

    验证工具链为 MoonBit 0.10.15+943464bfa（moon 0.1.20261012）。支持 wasm-gc、wasm、js、native；native 需要平台 C 编译工具，Windows 需要 MSVC C++ 工具与 Windows SDK。无第三方模块依赖，使用官方 core。

    在已有 MoonBit 工程内执行：

    moon add TerryFFF-jenius/moonarpa@0.1.0

    在使用者的 moon.pkg 中导入：

    import { "TerryFFF-jenius/moonarpa" @arpa }

    #快速开始

    以下代码可放入已有工程的可执行包；包设置 pkgtype(kind: "executable")。

    fn main raise {
      let model = @arpa.count_corpus([
        ["open", "the", "door"],
        ["open", "the", "door"],
        ["close", "the", "window"],
      ], 3).train_discount()
      let score = model.sentence(["open", "the", "door"])
      println(score.to_json().stringify())
      let text = model.to_arpa()
      let restored = @arpa.parse(text)
      assert_eq(restored.sentence(["open", "the", "door"]), score)
    }

    默认插入一个 BOS 历史和一个计分 EOS。因此上面的三个词产生四个预测事件。手工输入 <s>、</s> 会报错，边界通过 ScoreOptions 控制。可执行示例见 训练与恢复、语音重排、留出集评估。

    #API 与数值约定

    接口用途
    parse / Model::new / to_arpa严格解析、构造验证、确定性导出
    conditional(history, token)返回最长匹配阶数、回退次数、回退权重和 OOV 映射
    sentence / score_text / corpus / score_lines逐词评分与按预测事件加权的语料指标
    scorer / feed_batch / fork / finish独立增量状态、失败批次原子回滚、幂等结束
    predict / rank_candidates / rescore全词表前 k 项、候选集合排序、句子 n-best 重排
    Mixture::new同词表模型的概率混合，内部稳定 log-sum-exp
    count_corpus / Counts::merge有界计数与分片合并
    train_mle / train_discountMLE 与插值绝对折扣 ARPA 训练
    fit_vocabulary / transform训练集拟合固定词表与留出集 <unk> 映射
    audit_mass / audit_contexts / summary概率质量及模型结构诊断
    execute / handle可移植 JSON 任务引擎与成功/错误信封

    • 全部概率及回退权重使用 log10。负无穷表示不可能事件；正回退权重合法。模型的显式概率必须非正。合法 ARPA 不一定归一化，可通过 audit_mass 检查。
    • 缺失高阶 n-gram 时加入对应历史的回退权重，再移除最早的历史词；缺失权重按零处理。conditional 只使用最近 order-1 个历史词，但输入总量仍受预算约束。
    • OOV 默认映射到 <unk>；模型缺失该 unigram 或指定 Reject 时返回错误。已经映射成字面 <unk> 的输入不再计为新 OOV，MappedCorpus.replaced 记录预处理替换数。
    • 困惑度是 10^(-sum_log10 / predicted)；EOS 默认计入分母，BOS 不计入。没有预测事件返回 None；不可能事件得到无限困惑度。JSON 使用字符串 "Infinity"、"-Infinity"，无事件指标使用 null。
    • rescore 计算 lm_weight*LM + acoustic_weight*acoustic + length_weight*词数，并列按输入顺序；声学分数须由调用者换算到相容单位。零 LM 权重会跳过不可能似然项，避免 0 * -inf。
    • words 只按 Unicode 空白分割，不执行语言学分词、大小写或 Unicode 规范化。token 不能为空或包含空白、ASCII 控制字符、DEL。
    • 模型和词表拥有内部数据，公开数组返回副本。评分报告是调用方拥有的结果。混合模型的 matched_order=0 表示跨组件结果，不宣称单一匹配阶数。

    #运行工作流与 JSON 命令

    在本仓库根目录执行：

    moon run examples/speech
moon run examples/corpus
moon run examples/train
python scripts/run_request.py fixtures/train-request.json --target wasm-gc

    命令行包接收一个 JSON 参数并输出 {"ok":true,"result":...} 或 {"ok":false,"error":...}。Python 文件适配器检查信封后输出结果或以异常退出。大请求应直接调用库里的 execute，因为操作系统的 argv 上限可能小于模型预算。完整请求字段和例子见 JSON 协议。

    #资源限制与错误

    Limits 默认限制：最高八阶、20 万记录、1,600 万字符串单位、100 万输入 token、200 万工作事件。可以传入更低预算。受整数安全上限约束，最大值依次为 8 / 100 万 / 2.56 亿 / 1,000 万 / 1 亿。max_text 使用 MoonBit 的 String.length() 单位，不承诺 UTF-8 字节数或精确进程内存上限。

    声明记录数、输入长度、统计工作、候选扫描和语料累积都受到检查；排序仍有 O(R log R) 成本。计数训练在内存进行，没有磁盘外排序。ArpaError 分为 Invalid、Budget、Missing。ARPA 严格要求连续阶数声明、升序 section、精确计数、结束标记；支持空行、行首 # 注释、CRLF 与 -inf，不支持行尾注释及无词表符号的高阶记录。

    #检查与复现

    moon info
moon fmt --check
moon check --deny-warn --target wasm-gc
moon test --deny-warn --target wasm-gc
moon build --deny-warn --target wasm-gc
python scripts/check_project.py
python scripts/verify_oracle.py --target wasm-gc
python scripts/verify_reference.py --target wasm-gc

    将 target 换成 wasm、js、native 可复现其他后端。CI 对四目标执行严格检查/测试/构建、三个工作流和独立 Python oracle。参考测试按固定 SHA 从 KenLM 官方仓库读取测试数据并校验 SHA256，需要网络；也可用 --fixture PATH 指向同摘要的已下载数据。没有执行 KenLM 二进制，不能视为全量 KenLM 兼容认证。

    独立 oracle 以线性概率和独立 Counter 计数核对 25 组训练场景、1,225 个预测事件；KenLM 检查 10 个公开数值与 22 个预测事件。MoonBit 测试覆盖八种模型阶数、空语料、OOV、多层回退、正回退、概率质量、输入预算、溢出和规范化往返。详细设计见 设计、限制与安全、查重证据、版本记录。这些是软件验证证据，不代表赛事官方验收或实际生产性能。

    #许可与 AI 使用

    原创 MoonBit 与辅助脚本采用 MIT，作者 TerryFFF-jenius。本项目参考 ARPA/KenLM 的格式和评分行为，未复制其引擎代码；KenLM 测试数据仅在验证时从上游读取，未包含在发布包。上游许可及边界见 THIRD_PARTY.md。

    实现、测试、文档和交付由 OpenAI Codex 协助完成。AI 输出经过编译、测试、独立数值核验与接口审查；未声称用户已有生产部署或性能基准。包不包含任何私有参赛文书或联系方式。

    ArpaError

    source
    pub(all) suberror ArpaError {
      Invalid(String)
      Budget(String)
      Missing(String)
    } derive(
    Debug    )

    CorpusScore

    source
    pub(all) struct CorpusScore {
      sentences : Int
      predicted : Int
      oov_count : Int
      impossible : Int
      log_probability : Double
      sentence_scores : Array[SentenceScore]
    } derive(Eq, 
    Debug    )

    CorpusScore::cross_entropy_bits

    source
    fn CorpusScore::cross_entropy_bits(self : CorpusScore) -> Double?

    CorpusScore::equal

    source
    fn CorpusScore::equal(CorpusScore, CorpusScore) -> Bool

    CorpusScore::not_equal

    source
    fn CorpusScore::not_equal(x : CorpusScore, y : CorpusScore) -> Bool

    CorpusScore::oov_rate

    source
    fn CorpusScore::oov_rate(self : CorpusScore) -> Double?

    CorpusScore::perplexity

    source
    fn CorpusScore::perplexity(self : CorpusScore) -> Double?

    CorpusScore::to_json

    source

    CorpusScore::to_repr

    source

    CountGram

    source
    pub(all) struct CountGram {
      tokens : Array[String]
      count : Int
    } derive(Eq, 
    Debug    )

    CountGram::equal

    source
    fn CountGram::equal(CountGram, CountGram) -> Bool

    CountGram::not_equal

    source
    fn CountGram::not_equal(x : CountGram, y : CountGram) -> Bool

    CountGram::to_repr

    source

    Counts

    source
    pub struct Counts {
      order : Int
      entries : Array[CountGram]
      index : Map[String, Int]
      sentences : Int
      observed : Int
      limits : Limits
    }

    Counts::continuations

    source
    fn Counts::continuations(self : Counts, history : Array[String]) -> Array[CountGram] raise ArpaError

    Sorted continuation counts for inspecting an observed history.

    Counts::count

    source
    fn Counts::count(self : Counts, tokens : Array[String]) -> Int

    Counts::merge

    source
    fn Counts::merge(self : Counts, other : Counts) -> Counts raise ArpaError

    Merge independent corpus shards without retaining raw training sentences. Counts use identical boundary rules; each shard's sentence boundaries remain.

    Counts::order

    source
    fn Counts::order(self : Counts) -> Int

    Counts::records

    source
    fn Counts::records(self : Counts) -> Array[CountGram]

    Counts::sentence_count

    source
    fn Counts::sentence_count(self : Counts) -> Int

    Counts::token_count

    source
    fn Counts::token_count(self : Counts) -> Int

    Counts::train_discount

    source
    fn Counts::train_discount(self : Counts, discount? : Double, unigram_prior? : Double) -> Model raise ArpaError

    Interpolated absolute discount: seen estimates include lower-order mass; absent events receive lambda times the suffix estimate through ARPA backoff.

    Counts::train_mle

    source
    fn Counts::train_mle(self : Counts, unigram_prior? : Double) -> Model raise ArpaError

    Maximum likelihood above the unigram level. Observed histories have zero unseen mass; unseen histories fall back. Additive prior applies to unigrams.

    Counts::vocabulary

    source
    fn Counts::vocabulary(self : Counts) -> Array[String]

    FittedVocabulary

    source
    pub struct FittedVocabulary {
      words : Array[String]
      membership : Map[String, Bool]
      training_tokens : Int
      removed_types : Int
      limits : Limits
    }

    FittedVocabulary::contains

    source
    fn FittedVocabulary::contains(self : FittedVocabulary, word : String) -> Bool

    FittedVocabulary::map_token

    source
    fn FittedVocabulary::map_token(self : FittedVocabulary, word : String) -> String raise ArpaError

    FittedVocabulary::removed_types

    source
    fn FittedVocabulary::removed_types(self : FittedVocabulary) -> Int

    FittedVocabulary::training_tokens

    source
    fn FittedVocabulary::training_tokens(self : FittedVocabulary) -> Int

    FittedVocabulary::transform

    source
    fn FittedVocabulary::transform(self : FittedVocabulary, sentences : Array[Array[String]]) -> MappedCorpus raise ArpaError

    FittedVocabulary::words

    source
    fn FittedVocabulary::words(self : FittedVocabulary) -> Array[String]

    Gram

    source
    pub(all) struct Gram {
      tokens : Array[String]
      log_probability : Double
      log_backoff : Double?
    } derive(Eq, 
    Debug    )

    A single ARPA record; probability and optional backoff use base-10 logarithms.

    Gram::equal

    source
    fn Gram::equal(Gram, Gram) -> Bool

    Gram::not_equal

    source
    fn Gram::not_equal(x : Gram, y : Gram) -> Bool

    Gram::to_repr

    source

    Hypothesis

    source
    pub(all) struct Hypothesis {
      id : String
      tokens : Array[String]
      acoustic_score : Double
    } derive(Eq, 
    Debug    )

    Hypothesis::equal

    source
    fn Hypothesis::equal(Hypothesis, Hypothesis) -> Bool

    Hypothesis::not_equal

    source
    fn Hypothesis::not_equal(x : Hypothesis, y : Hypothesis) -> Bool

    Hypothesis::to_repr

    source

    Limits

    source
    pub(all) struct Limits {
      max_order : Int
      max_records : Int
      max_text : Int
      max_tokens : Int
      max_work : Int
    } derive(Eq, 
    Debug    )

    Limits apply before allocations or bounded inner loops.

    Limits::default

    source
    fn Limits::default() -> Limits

    Limits::equal

    source
    fn Limits::equal(Limits, Limits) -> Bool

    Limits::not_equal

    source
    fn Limits::not_equal(x : Limits, y : Limits) -> Bool

    Limits::to_repr

    source

    MappedCorpus

    source
    pub(all) struct MappedCorpus {
      sentences : Array[Array[String]]
      tokens : Int
      replaced : Int
    } derive(Eq, 
    Debug    )

    MappedCorpus::equal

    source

    MappedCorpus::not_equal

    source
    fn MappedCorpus::not_equal(x : MappedCorpus, y : MappedCorpus) -> Bool

    MappedCorpus::to_repr

    source

    MassAudit

    source
    pub(all) struct MassAudit {
      history : Array[String]
      mass : Double
      deviation : Double
      normalized : Bool
      zero_events : Int
      backed_off_events : Int
    } derive(Eq, 
    Debug    )

    MassAudit::equal

    source
    fn MassAudit::equal(MassAudit, MassAudit) -> Bool

    MassAudit::not_equal

    source
    fn MassAudit::not_equal(x : MassAudit, y : MassAudit) -> Bool

    MassAudit::to_json

    source

    MassAudit::to_repr

    source

    Mixture

    source
    pub struct Mixture {
      models : Array[Model]
      weights : Array[Double]
      limits : Limits
    }

    A probability mixture of models with exactly the same vocabulary.

    Mixture::conditional

    source
    fn Mixture::conditional(self : Mixture, history : Array[String], token : String, oov? : OovPolicy) -> Double raise ArpaError

    Stable log10-sum-exp; mixture is of probabilities, not logarithms.

    Mixture::new

    source
    fn Mixture::new(models : Array[Model], weights : Array[Double], limits? : Limits) -> Mixture raise ArpaError

    Mixture::sentence

    source
    fn Mixture::sentence(self : Mixture, tokens : Array[String], options? : ScoreOptions) -> SentenceScore raise ArpaError

    Mixture::weights

    source
    fn Mixture::weights(self : Mixture) -> Array[Double]

    Model

    source
    pub struct Model {
      order : Int
      grams : Array[Gram]
      index : Map[String, Int]
      vocabulary : Array[String]
      vocabulary_set : Map[String, Bool]
      limits : Limits
    }

    Owned validated records; callers cannot mutate the internal indices.

    Model::audit_contexts

    source
    fn Model::audit_contexts(self : Model, maximum : Int, tolerance? : Double) -> Array[MassAudit] raise ArpaError

    Audits a deterministic bounded set of observed histories and the empty one. The caller chooses a cap because exhaustive V^order enumeration is unbounded.

    Model::audit_mass

    source
    fn Model::audit_mass(self : Model, history : Array[String], tolerance? : Double) -> MassAudit raise ArpaError

    Sum the complete vocabulary with compensated summation. ARPA syntax alone does not guarantee normalized conditional distributions.

    Model::conditional

    source
    fn Model::conditional(self : Model, history : Array[String], token : String, oov? : OovPolicy) -> TokenScore raise ArpaError

    Longest matching suffix; each missing ngram contributes its history backoff.

    Model::contains

    source
    fn Model::contains(self : Model, token : String) -> Bool

    Model::corpus

    source
    fn Model::corpus(self : Model, sentences : Array[Array[String]], options? : ScoreOptions) -> CorpusScore raise ArpaError

    Aggregates token likelihoods, never an arithmetic mean of sentence perplexities.

    Model::counts

    source
    fn Model::counts(self : Model) -> Array[Int]

    Model::new

    source
    fn Model::new(order : Int, records : Array[Gram], limits? : Limits) -> Model raise ArpaError

    Model::order

    source
    fn Model::order(self : Model) -> Int

    Model::predict

    source
    fn Model::predict(self : Model, history : Array[String], k : Int, include_unknown? : Bool) -> Array[Prediction] raise ArpaError

    Sorted top-k vocabulary continuations; probabilities are not renormalized.

    Model::rank_candidates

    source
    fn Model::rank_candidates(self : Model, history : Array[String], tokens : Array[String], oov? : OovPolicy) -> Array[Prediction] raise ArpaError

    Scores only a caller-supplied candidate set, deduplicating exact token strings.

    Model::record

    source
    fn Model::record(self : Model, tokens : Array[String]) -> Gram?

    Model::records

    source
    fn Model::records(self : Model) -> Array[Gram]

    Model::rescore

    source
    fn Model::rescore(self : Model, hypotheses : Array[Hypothesis], options? : RescoreOptions) -> Array[RankedHypothesis] raise ArpaError

    Acoustic scores must already use compatible units (typically log10). Zero LM weight disables the LM term even for impossible sentences.

    Model::score_lines

    source
    fn Model::score_lines(self : Model, text : String, options? : ScoreOptions) -> CorpusScore raise ArpaError

    Each physical line is one sentence, including empty lines between newlines. A terminal newline terminates the last line and does not add an empty sentence.

    Model::score_text

    source
    fn Model::score_text(self : Model, text : String, options? : ScoreOptions) -> SentenceScore raise ArpaError

    Model::scorer

    source
    fn Model::scorer(self : Model, options? : ScoreOptions) -> Scorer raise ArpaError

    Model::sentence

    source
    fn Model::sentence(self : Model, tokens : Array[String], options? : ScoreOptions) -> SentenceScore raise ArpaError

    Inputs are already tokenized. BOS seeds history but is never predicted.

    Model::size

    source
    fn Model::size(self : Model) -> Int

    Model::summary

    source
    fn Model::summary(self : Model) -> ModelSummary

    Model::to_arpa

    source
    fn Model::to_arpa(self : Model) -> String raise ArpaError

    Deterministic section/lexicographic order; numeric precision is not truncated.

    Model::vocabulary

    source
    fn Model::vocabulary(self : Model) -> Array[String]

    ModelSummary

    source
    pub(all) struct ModelSummary {
      order : Int
      records : Int
      vocabulary : Int
      counts : Array[Int]
      impossible_records : Int
      positive_backoffs : Int
      has_unknown : Bool
      has_bos : Bool
      has_eos : Bool
    } derive(Eq, 
    Debug    )

    ModelSummary::equal

    source

    ModelSummary::not_equal

    source
    fn ModelSummary::not_equal(x : ModelSummary, y : ModelSummary) -> Bool

    ModelSummary::to_json

    source

    ModelSummary::to_repr

    source

    OovPolicy

    source
    pub(all) enum OovPolicy {
      MapToUnknown
      Reject
    } derive(Eq, 
    Debug    )

    OovPolicy::equal

    source
    fn OovPolicy::equal(OovPolicy, OovPolicy) -> Bool

    OovPolicy::not_equal

    source
    fn OovPolicy::not_equal(x : OovPolicy, y : OovPolicy) -> Bool

    OovPolicy::to_repr

    source

    Prediction

    source
    pub(all) struct Prediction {
      token : String
      log_probability : Double
      matched_order : Int
    } derive(Eq, 
    Debug    )

    Prediction::equal

    source
    fn Prediction::equal(Prediction, Prediction) -> Bool

    Prediction::not_equal

    source
    fn Prediction::not_equal(x : Prediction, y : Prediction) -> Bool

    Prediction::to_json

    source

    Prediction::to_repr

    source

    RankedHypothesis

    source
    pub(all) struct RankedHypothesis {
      id : String
      input_index : Int
      score : Double
      language : SentenceScore
    } derive(Eq, 
    Debug    )

    RankedHypothesis::equal

    source

    RankedHypothesis::not_equal

    source
    fn RankedHypothesis::not_equal(x : RankedHypothesis, y : RankedHypothesis) -> Bool

    RankedHypothesis::to_json

    source

    RankedHypothesis::to_repr

    source

    RescoreOptions

    source
    pub(all) struct RescoreOptions {
      lm_weight : Double
      acoustic_weight : Double
      length_weight : Double
      scoring : ScoreOptions
    } derive(Eq, 
    Debug    )

    RescoreOptions::default

    source

    RescoreOptions::equal

    source

    RescoreOptions::not_equal

    source
    fn RescoreOptions::not_equal(x : RescoreOptions, y : RescoreOptions) -> Bool

    RescoreOptions::to_repr

    source

    ScoreOptions

    source
    pub(all) struct ScoreOptions {
      bos : Bool
      eos : Bool
      oov : OovPolicy
    } derive(Eq, 
    Debug    )

    ScoreOptions::default

    source
    fn ScoreOptions::default() -> ScoreOptions

    ScoreOptions::equal

    source

    ScoreOptions::not_equal

    source
    fn ScoreOptions::not_equal(x : ScoreOptions, y : ScoreOptions) -> Bool

    ScoreOptions::to_repr

    source

    Scorer

    source
    pub struct Scorer {
      model : Model
      options : ScoreOptions
      history : Array[String]
      results : Array[TokenScore]
      total : Double
      text_units : Int
      oovs : Int
      closed : Bool
    }

    A sentence-scoped incremental scorer. Failed batches leave the state unchanged.

    Scorer::feed

    source
    fn Scorer::feed(self : Scorer, token : String) -> TokenScore raise ArpaError

    Scorer::feed_batch

    source
    fn Scorer::feed_batch(self : Scorer, tokens : Array[String]) -> SentenceScore raise ArpaError

    Scorer::finish

    source
    fn Scorer::finish(self : Scorer) -> SentenceScore raise ArpaError

    Scorer::fork

    source
    fn Scorer::fork(self : Scorer) -> Scorer

    Scorer::history

    source
    fn Scorer::history(self : Scorer) -> Array[String]

    Scorer::is_closed

    source
    fn Scorer::is_closed(self : Scorer) -> Bool

    Scorer::reset

    source
    fn Scorer::reset(self : Scorer) -> Unit

    Scorer::snapshot

    source
    fn Scorer::snapshot(self : Scorer) -> SentenceScore

    SentenceScore

    source
    pub(all) struct SentenceScore {
      log_probability : Double
      predicted : Int
      oov_count : Int
      tokens : Array[TokenScore]
    } derive(Eq, 
    Debug    )

    SentenceScore::cross_entropy_bits

    source
    fn SentenceScore::cross_entropy_bits(self : SentenceScore) -> Double?

    Cross-entropy in bits per predicted event; None means no predicted events.

    SentenceScore::equal

    source

    SentenceScore::not_equal

    source
    fn SentenceScore::not_equal(x : SentenceScore, y : SentenceScore) -> Bool

    SentenceScore::perplexity

    source
    fn SentenceScore::perplexity(self : SentenceScore) -> Double?

    SentenceScore::to_json

    source

    SentenceScore::to_repr

    source

    TokenScore

    source
    pub(all) struct TokenScore {
      token : String
      mapped : String
      log_probability : Double
      matched_order : Int
      backoff_steps : Int
      backoff_log : Double
      oov : Bool
    } derive(Eq, 
    Debug    )

    TokenScore::equal

    source
    fn TokenScore::equal(TokenScore, TokenScore) -> Bool

    TokenScore::not_equal

    source
    fn TokenScore::not_equal(x : TokenScore, y : TokenScore) -> Bool

    TokenScore::to_json

    source

    TokenScore::to_repr

    source

    VocabularyOptions

    source
    pub(all) struct VocabularyOptions {
      min_count : Int
      max_vocabulary : Int
    } derive(Eq, 
    Debug    )

    VocabularyOptions::default

    source

    VocabularyOptions::equal

    source

    VocabularyOptions::not_equal

    source
    fn VocabularyOptions::not_equal(x : VocabularyOptions, y : VocabularyOptions) -> Bool

    VocabularyOptions::to_repr

    source

    count_corpus

    source
    fn count_corpus(sentences : Array[Array[String]], order : Int, limits? : Limits) -> Counts raise ArpaError

    Counts one BOS history and one predicted EOS for every sentence. BOS is not a predicted event and gets a zero-count vocabulary entry.

    execute

    source
    fn execute(request : String, limits? : Limits) -> 
    Json     raise ArpaError

    Portable, bounded JSON task engine. There is no filesystem or network access. Unknown fields fail instead of silently changing a user's scoring options.

    fit_vocabulary

    source
    fn fit_vocabulary(sentences : Array[Array[String]], options? : VocabularyOptions, limits? : Limits) -> FittedVocabulary raise ArpaError

    Fit on training data only; reuse the resulting mapping for held-out data. Count ties are resolved lexically and the three reserved symbols always exist.

    handle

    source
    fn handle(request : String, limits? : Limits) -> String

    A stable envelope for CLI hosts. Errors contain no partial successful result.

    parse

    source
    fn parse(text : String, limits? : Limits) -> Model raise ArpaError

    Strict ARPA text reader with declared count and section-order verification.

    words

    source
    fn words(text : String) -> Array[String]

    Splits on Unicode whitespace without linguistic tokenization or normalization.

    Source Files