moonabnf

    Byte-oriented ABNF grammar AST, validation, budgeted recognition and input derivation trees

    abnf
    rfc5234
    rfc7405
    parser
    Download zip
    Version
    0.3.0
    License
    MIT
    Last updated
    10 days ago
    Downloads
    137

    No dependencies

    #MoonABNF

    CI Fresh clone License: MIT

    用 MoonBit 写的字节级 ABNF 库:直接拿 RFC 里的 ABNF 文法校验协议字段,不用手写解析器。

    文法文本 → 文法 AST → 编译校验 → 整段识别 / 输入派生树 → 按规则名取出原字节字段。 实现依据 RFC 5234(ABNF)与 RFC 7405(大小写敏感字符串)。

    适合:协议字段离线诊断、配置格式校验、二进制 fixture 回归、把 RFC 文法变成可执行的测试。

    本地开发候选为0.3.0(有界输入失败诊断),尚未公开或发布;当前可安装版本仍为0.2.1。候选变更与发布条件。

    #快速开始

    需要 MoonBit 工具链(CI 锁定 moonc v0.10.14+7d59c7ec9,见 TOOLCHAIN.lock)和 Node.js。

    git clone https://github.com/sundaysebasidian-byte/MoonABNF.git cd MoonABNF moon test # 全部单元测试 moon run examples --target js # 协议 / 配置 / 二进制三个小场景 moon run examples/protocols/cmd --target js # URI 与 Cookie:输入树和字段提取

    在自己的项目里使用:

    moon add sundaysebasidian-byte/moonabnf@0.2.1

    0.2.1 保持 0.2.0 公共解析 API,补充可复现 registry 协议消费者、独立 AI 复审与五进程测量说明。JSON 适配器仍是集成示例。 版本内容及验证边界见 0.2.1 说明;实际发布状态以 Mooncakes 为准。

    // moon.pkg
    import { "sundaysebasidian-byte/moonabnf" @abnf }

    0.2.0 已发布到 Mooncakes:包含 parse / compile / recognize 以及输入派生树与字段提取(parse_input、rule_nodes_in、extract)。全新 registry 消费者已验证 Wasm、Wasm GC、JS 和 native;JS JSON 协议适配为需复制的 集成示例,使用公开核心 API 和 examples/protocols 文法子包,并非根包 JSON 导出。

    #用法

    #1. 校验一个字段

    match @abnf.compile(b"field = %i\"Trace\" \":\" SP 1*8HEXDIG CRLF\r\n") {
    Err(d) => println("\{d.code} at line \{d.line}, column \{d.column}")
    Ok(program) => {
    let r = program.recognize("field", b"TRACE: aF12\r\n")
    assert_true(r.status is @abnf.Matched)
    }
    }

    文法写错时得到带错误码、字节偏移和行列号的 Diagnostic,例如 undefined-rule、left-recursion、duplicate-rule。

    #2. 取出字段的原始字节

    let program = @abnf.compile(
    b"record = key \":\" value\r\nkey = 1*ALPHA\r\nvalue = 1*DIGIT\r\n",
    ).unwrap()
    let input = b"Age:42"
    let report = program.parse_input("record", input)
    match report.status {
    @abnf.Matched => {
    let tree = report.tree.unwrap()
    let ids = tree.rule_nodes_in("value", tree.root).unwrap()
    assert_eq(tree.extract(input, ids[0]), Some(b"42"))
    }
    @abnf.NoMatch => println("no-match at byte \{report.offset}")
    @abnf.BudgetExceeded(why) => println("indeterminate: \{why}")
    _ => ()
    }

    本页代码片段都在 readme_test.mbt 中作为测试运行;完整示例见 examples/input_tree/main.mbt。

    #3. 用真实 RFC 文法

    examples/protocols 直接使用 RFC 3986(URI)和 RFC 6265(Cookie)的 ABNF,输出输入树前三层和各组件的字节跨度:

    URI components (syntax only, no percent decoding): scheme=[0,5) raw=b"https" host=[8,21) raw=b"[2001:db8::1]" port=[22,26) raw=b"8443" ... Cookie pairs (duplicates and quotes preserved): cookie-name=[0,7) raw=b"session" cookie-value=[8,11) raw=b"abc" ... Invalid Cookie: no-match at byte 11 Limited Cookie: indeterminate (tree-nodes), tree absent=true

    另有 RFC 9110 ETag(examples/http_etag)与 RFC 9112 请求行(examples/request_line)示例,输出都与仓库里的 expected_output.txt 逐字节比对。

    实际应用:离线采集记录导入门禁复用独立URI/Cookie消费者, 把规范原例子转换为带字节span的JSON字段;用不同退出码区分语法拒绝、预算未判定和配置错误。 只有两字段均匹配才导出数据,并与仅识别的现有API作具体对照;不宣称完整HTTP支持或生产采用。

    #4. 命令行

    scripts/cli.sh check fixtures/protocol.abnf scripts/cli.sh match fixtures/protocol.abnf field fixtures/protocol-valid.bin

    输出 JSON,退出码可直接用于脚本门禁。

    #规范覆盖

    RFC 5234 / 7405支持
    规则定义 =、增量替代 =/,规则名大小写不敏感✅
    替代 /、串联、分组 ( )、可选 [ ]✅
    重复 n*m、*m、n*、*、n✅
    数值 %b %d %x:单值、点连接序列、区间✅
    字符串:默认大小写不敏感,%i / %s(RFC 7405)✅
    注释、CRLF 行结构✅
    附录 B 核心规则 ALPHA … WSP(16 条)✅ 内置,可被同名规则覆盖
    prose-val <...>解析进 AST;编译时报 prose-unsupported

    编译阶段检查未定义规则、重复定义、不产生任何串的规则,以及左递归 / 可空前缀循环。

    #设计取舍

    • 两棵树分开:文法 AST 描述文法本身,输入派生树描述某段输入怎样被匹配。
    • ABNF 不是 PEG:替代分支会保留全部可能的结束位置,不会因为先匹配到短的分支就错判。
    • 资源有预算:输入长度、工作量、状态数、深度、树节点都有上限。预算用完返回 BudgetExceeded(未判定),不会被当成“不匹配”。
    • 偏移是字节:跨度是半开字节区间,extract 从调用方保存的原始输入里复制,不做 URI 解码或去引号。
    • 不做:Unicode 数值终端、错误恢复、代码生成、流式输入、歧义全枚举,以及 URI/Cookie 的语义层规则。

    详细契约见 边界说明 和 输入树上手,公开接口见 pkg.generated.mbti。

    #测试

    • 单元与场景测试:解析器、编译器、识别器、输入树、边界、鲁棒性(1024 个变异文法、预算终止)。
    • 差分测试:随机生成 80 个文法、5440 次比对,结果与独立的正则 oracle 一致(scripts/differential.py)。
    • 真实协议语料:URI、Cookie、ETag、HTTP 请求行。
    • CI:ci.yml 在官方 SDK 上跑完整验收(JS / Wasm / Wasm GC / native 的 check、build、test,示例、CLI、差分、下游包消费);fresh-clone.yml 模拟新用户 git clone 后直接 moon check、moon test 和本页的快速开始。

    本地完整验收:python3 scripts/acceptance.py --output /tmp/moonabnf-evidence。 各轮验证的命令、版本和结果记录在 docs/VERIFICATION_20261022.zh-CN.md 和 evidence/。

    #目录

    路径内容
    parser.mbt types.mbtABNF 文本 → 文法 AST
    compiler.mbt analysis.mbt core.mbt规则合并、核心规则、引用 / 生产性 / 左递归检查
    recognizer.mbt有预算的整段识别
    input_tree.mbt input_query.mbt输入派生树、子树查询、字段提取
    cmd/mainJSON 命令行
    examples/可运行示例与黄金输出
    fixtures/测试文法与 RFC 文法语料(来源见 fixtures/protocols/SOURCES.md)

    #许可与来源

    代码 MIT(LICENSE)。从 RFC 摘录的 URI / Cookie / HTTP 文法保留原有归属,不改称 MIT 原创,出处与哈希见 SOURCES.md。 开发中 AI 的参与方式见 AI_USAGE.md,设计说明见 docs/UNDERSTANDING.md。

    #未发布源码:有界输入失败解释

    新开发入口 program.diagnose_input(rule, input) 保留原 MatchReport,仅对 NoMatch 作额外有界重放。failure 含局部最远失败字节、actual 字节整数或 EOF(None)、有限期望字节范围/ASCII 大小写语义或 EOF,以及可确定的原文法 terminal expression 起点 grammar_offset;内建 core terminal 和 EOF 的位置为 None。 它不是唯一修复建议,也不枚举全部失败路径。成功或 BudgetExceeded 不产生失败解释。

    diagnostic_limits={ max_work: 500000, max_expectations: 16 } 独立限制额外重放; 工作包含重放、期望收集及去重,期望硬上限128。explanation_status 和 failure.truncated 区分解释预算不足/截断与原匹配判定;原接受集合、树与匹配预算不变。 InputDiagnosticReport.to_json() 只序列化解释,不另作接受判定。

    源码 CLI 可用 moon run cmd/main --target js -- diagnose grammar.abnf rule input.bin [work-budget]; 原 match 输出不变。JSON 案例须用独立 main.diagnostics.mbt.in 扩展模板, python3 scripts/protocol_record_test.py --diagnostics --output /new/external/evidence 验证后,可在该消费者的 --json URI COOKIE [node-budget] 最后加 --diagnostics。 未加标志时黄金及退出0/1/3/2不变。原模板和 registry 消费者保持兼容已发布0.2.1; 这些新 API/扩展仍是本地开发源码,不属于已发布的0.2.1。

    新手可先运行 moon run examples/diagnostics --target js:URI %GG、Cookie 空格、 截断与尾随空格四例会显示原字节上下文、实际字节/EOF和局部期望,再对人工选择的合法 修订重新解析并提取字段。它不自动更改输入,也不把预算不足当拒绝;示例使用合成协议字段。

    Diagnostic

    pub(all) struct Diagnostic {
    code : String
    message : String
    offset : Int
    line : Int
    column : Int
    } derive(Eq,
    Debug
    )

    Source positions are zero-based byte offsets; line/column are one-based.

    Diagnostic::equal

    fn Diagnostic::equal(Diagnostic, Diagnostic) -> Bool

    Diagnostic::not_equal

    fn Diagnostic::not_equal(x : Diagnostic, y : Diagnostic) -> Bool

    ExpectedInput

    pub(all) struct ExpectedInput {
    expectation : InputExpectation
    grammar_offset : Int?
    } derive(Eq,
    Debug
    )

    grammar_offset is the terminal expression's original grammar byte offset. None denotes implicit core terminals or an end-of-input requirement.

    ExpectedInput::equal

    ExpectedInput::not_equal

    fn ExpectedInput::not_equal(x : ExpectedInput, y : ExpectedInput) -> Bool

    Expr

    pub(all) enum Expr {
    Ref(String, Int)
    Literal(Bytes, Bool, Int)
    Range(Int, Int, Int)
    Sequence(Array[Expr], Int)
    Choice(Array[Expr], Int)
    Repeat(Int, Int?, Expr, Int)
    Prose(String, Int)
    } derive(Eq,
    Debug
    )

    Public AST. Repeat maximum None means unbounded; positions refer to source.

    Expr::equal

    fn Expr::equal(Expr, Expr) -> Bool

    Expr::not_equal

    fn Expr::not_equal(x : Expr, y : Expr) -> Bool

    Expr::to_repr

    Grammar

    pub(all) struct Grammar {
    rules : Array[Rule]
    } derive(Eq,
    Debug
    )

    Grammar::equal

    fn Grammar::equal(Grammar, Grammar) -> Bool

    Grammar::not_equal

    fn Grammar::not_equal(x : Grammar, y : Grammar) -> Bool

    Grammar::to_repr

    InputDiagnosticLimits

    pub(all) struct InputDiagnosticLimits {
    max_work : Int
    max_expectations : Int
    } derive(Eq,
    Debug
    )

    InputDiagnosticLimits::equal

    InputDiagnosticLimits::not_equal

    InputDiagnosticReport

    pub(all) struct InputDiagnosticReport {
    report : MatchReport
    failure : InputFailure?
    explanation_status : String
    explanation_work : Int
    } derive(Eq,
    Debug
    )

    match is the unchanged recognition result. explanation_status describes only the separately budgeted explanatory replay (not another acceptance verdict).

    InputDiagnosticReport::equal

    InputDiagnosticReport::not_equal

    InputDiagnosticReport::to_json

    Optional adapter representation, separate from the existing match JSON schema.

    InputExpectation

    pub(all) enum InputExpectation {
    ByteRange(Int, Int, Bool)
    EndOfInput
    } derive(Eq,
    Debug
    )

    ByteRange's third value denotes ASCII case-insensitive matching, not Unicode.

    InputExpectation::equal

    InputExpectation::not_equal

    fn InputExpectation::not_equal(x : InputExpectation, y : InputExpectation) -> Bool

    InputFailure

    pub(all) struct InputFailure {
    offset : Int
    actual : Int?
    expected : Array[ExpectedInput]
    truncated : Bool
    } derive(Eq,
    Debug
    )

    InputFailure::equal

    InputFailure::not_equal

    fn InputFailure::not_equal(x : InputFailure, y : InputFailure) -> Bool

    InputNode

    pub(all) struct InputNode {
    kind : InputNodeKind
    start : Int
    end : Int
    source_offset : Int
    children : Array[Int]
    } derive(Eq,
    Debug
    )

    Half-open input byte span. source_offset refers to the grammar expression; implicit core expressions have synthetic offset 0. Children are node IDs.

    InputNode::equal

    fn InputNode::equal(InputNode, InputNode) -> Bool

    InputNode::not_equal

    fn InputNode::not_equal(x : InputNode, y : InputNode) -> Bool

    InputNodeKind

    pub(all) enum InputNodeKind {
    RuleNode(String)
    LiteralNode
    RangeNode
    SequenceNode
    ChoiceNode(Int)
    RepeatNode(Int)
    } derive(Eq,
    Debug
    )

    A witness of an input derivation, distinct from the grammar Expr AST.

    InputNodeKind::equal

    InputNodeKind::not_equal

    fn InputNodeKind::not_equal(x : InputNodeKind, y : InputNodeKind) -> Bool

    InputTree

    pub(all) struct InputTree {
    nodes : Array[InputNode]
    root : Int
    } derive(Eq,
    Debug
    )

    Flat, reachable-only tree; root is always 0. No input bytes are copied.

    InputTree::equal

    fn InputTree::equal(InputTree, InputTree) -> Bool

    InputTree::extract

    fn InputTree::extract(self : InputTree, input : Bytes, node : Int) -> Bytes?

    Copies raw bytes from the caller's original input; does not decode UTF-8, URI escapes or quoted strings. Input identity remains caller responsibility. Invalid IDs/spans or an input shorter than the span return None. A different input of sufficient length is accepted, even if its bytes or total length differ. Use the exact original Bytes; this is not validation.

    InputTree::not_equal

    fn InputTree::not_equal(x : InputTree, y : InputTree) -> Bool

    InputTree::rule_nodes

    fn InputTree::rule_nodes(self : InputTree, name : String) -> Array[Int]

    Matching rule nodes in the tree's stable storage order (breadth-first for parse_input results). Rule names are ASCII case-insensitive as in ABNF. Does not traverse caller-mutable children, so malformed cycles cannot loop.

    InputTree::rule_nodes_in

    fn InputTree::rule_nodes_in(self : InputTree, name : String, root : Int) -> Array[Int]?

    Named nodes within a subtree, including root, in depth-first pre-order. Unlike rule_nodes' storage order, this follows the derivation's child order: outer captures precede inner captures, and siblings follow input order. Returns None for an invalid ID, span, arity, cycle or shared descendant in the selected subtree. Unreachable nodes outside it are not inspected. Iterative; storage is proportional to visited descendants, not the entire tree. Never schedules a child twice. This checks structure, not grammar membership or original-input identity.

    test {
    let p = @moonabnf.compile(b"s = item item\r\nitem = DIGIT\r\n").unwrap()
    let tree = p.parse_input("s", b"42").tree.unwrap()
    let ids = tree.rule_nodes_in("item", tree.root).unwrap()
    assert_eq(ids.map(id => tree.extract(b"42", id)), [Some(b"4"), Some(b"2")])
    }

    InputTree::span

    fn InputTree::span(self : InputTree, node : Int) -> (Int, Int)?

    Half-open byte span, or None for an invalid node ID or malformed span. Zero-width spans are valid. This local check does not validate topology, the root, grammar membership or the span against an input length.

    InputTreeLimits

    pub(all) struct InputTreeLimits {
    matching : MatchLimits
    max_nodes : Int
    max_edges : Int
    } derive(Eq,
    Debug
    )

    Allocation budgets count all explored witnesses, paths and final compaction, not only the returned tree. Match limits retain their existing meaning.

    InputTreeLimits::equal

    InputTreeLimits::not_equal

    fn InputTreeLimits::not_equal(x : InputTreeLimits, y : InputTreeLimits) -> Bool

    InputTreeReport

    pub(all) struct InputTreeReport {
    status : MatchStatus
    tree : InputTree?
    offset : Int
    work : Int
    states : Int
    nodes : Int
    edges : Int
    } derive(Eq,
    Debug
    )

    tree is Some exactly when status is Matched. Budget failure discards trees.

    InputTreeReport::equal

    InputTreeReport::not_equal

    fn InputTreeReport::not_equal(x : InputTreeReport, y : InputTreeReport) -> Bool

    MatchLimits

    pub(all) struct MatchLimits {
    max_input_bytes : Int
    max_work : Int
    max_states : Int
    max_depth : Int
    } derive(Eq,
    Debug
    )

    Work counts expression dispatches, byte comparisons and endpoint operations.

    MatchLimits::equal

    fn MatchLimits::equal(MatchLimits, MatchLimits) -> Bool

    MatchLimits::not_equal

    fn MatchLimits::not_equal(x : MatchLimits, y : MatchLimits) -> Bool

    MatchReport

    pub(all) struct MatchReport {
    status : MatchStatus
    offset : Int
    work : Int
    states : Int
    } derive(Eq,
    Debug
    )

    offset is a zero-based furthest attempted byte; full match offset is length. Budget exhaustion is indeterminate, even if an explored branch matched.

    MatchReport::equal

    fn MatchReport::equal(MatchReport, MatchReport) -> Bool

    MatchReport::not_equal

    fn MatchReport::not_equal(x : MatchReport, y : MatchReport) -> Bool

    MatchStatus

    pub(all) enum MatchStatus {
    Matched
    NoMatch
    BudgetExceeded(String)
    UnknownRule
    InvalidLimits
    } derive(Eq,
    Debug
    )

    MatchStatus::equal

    fn MatchStatus::equal(MatchStatus, MatchStatus) -> Bool

    MatchStatus::not_equal

    fn MatchStatus::not_equal(x : MatchStatus, y : MatchStatus) -> Bool

    ParseLimits

    pub(all) struct ParseLimits {
    max_bytes : Int
    max_rules : Int
    max_nodes : Int
    max_depth : Int
    } derive(Eq,
    Debug
    )

    Hard implementation ceilings prevent caller budgets from risking the stack.

    ParseLimits::equal

    fn ParseLimits::equal(ParseLimits, ParseLimits) -> Bool

    ParseLimits::not_equal

    fn ParseLimits::not_equal(x : ParseLimits, y : ParseLimits) -> Bool

    Program

    pub struct Program {
    // private fields
    }

    Compiled rules are private. Each compile parses its own AST, so caller-owned mutable AST arrays cannot invalidate the compiled program.

    Program::diagnose_input

    fn Program::diagnose_input(self : Program, start : String, input : Bytes, limits? : MatchLimits, diagnostic_limits? : InputDiagnosticLimits) -> InputDiagnosticReport

    Explain a NoMatch using bounded local expectations at the farthest observed failed terminal/complete-prefix endpoint. Not a unique repair or full trace. Successful/unknown recognition is never overwritten by exploratory failure.

    Program::parse_input

    fn Program::parse_input(self : Program, start : String, input : Bytes, limits? : InputTreeLimits) -> InputTreeReport

    Parse the whole input into one deterministic derivation witness. All ABNF alternatives/endpoints are explored; source order breaks equal-endpoint ties. Repeats try counts ascending, retaining the first path at each exact count. This does not enumerate ambiguity or change recognize's budgets/behavior.

    test {
    let p = @moonabnf.compile(b"s = 1*DIGIT\r\n").unwrap()
    let report = p.parse_input("s", b"42")
    assert_true(report.status is @moonabnf.Matched)
    assert_eq(report.tree.unwrap().nodes[0].end, 2)
    }

    Program::recognize

    fn Program::recognize(self : Program, start : String, input : Bytes, limits? : MatchLimits) -> MatchReport

    Recognize the entire byte sequence, not just a prefix.

    Program::rule_names

    fn Program::rule_names(self : Program) -> Array[String]

    Deterministic inventory useful to grammar-checking tools.

    Rule

    pub(all) struct Rule {
    name : String
    expression : Expr
    incremental : Bool
    offset : Int
    } derive(Eq,
    Debug
    )

    Rule::equal

    fn Rule::equal(Rule, Rule) -> Bool

    Rule::not_equal

    fn Rule::not_equal(x : Rule, y : Rule) -> Bool

    Rule::to_repr

    compile

    fn compile(source : Bytes, limits? : ParseLimits, max_work? : Int) -> Result[Program, Diagnostic]

    Core rules are implicit defaults. Explicit '=' overrides a core default; '=/‘ may extend it. Duplicate explicit '=' and extension before base fail.

    default_input_diagnostic_limits

    fn default_input_diagnostic_limits() -> InputDiagnosticLimits

    default_input_tree_limits

    fn default_input_tree_limits() -> InputTreeLimits

    default_match_limits

    fn default_match_limits() -> MatchLimits

    default_parse_limits

    fn default_parse_limits() -> ParseLimits

    parse

    fn parse(source : Bytes, limits? : ParseLimits) -> Result[Grammar, Diagnostic]

    Strict grammar input: CRLF required, including the final rule terminator.