conllu

    CoNLL-U / Universal Dependencies toolchain: lossless parse and serialize, spec validation, tree queries, UAS/LAS evaluation, corpus statistics.

    conllu
    universal-dependencies
    nlp
    dependency-parsing
    treebank
    Download zip
    Author
    Version
    0.1.0
    License
    Apache-2.0
    Last updated
    18 hours ago
    Downloads
    25

    #PaperY0/conllu

    CI mooncakes license

    MoonBit 里的 CoNLL-U / Universal Dependencies 工具链:无损解析与序列化、规范校验、依存树查询、UAS/LAS 评测、语料统计,外加一个可直接跑的 CLI。

    CoNLL-U 是 UD 树库(Universal Dependencies 项目,200+ 语言树库)的唯一交换格式:每句 10 列制表符分隔,句子之间空行分隔,# 开头是注释,ID 可以是 12(普通词)、5-6(多词 token)、5.1(空节点)。生态里做分词、词性的包已经不少,但句法层(HEAD/DEPREL 树、投射性、UAS/LAS)此前是空白,这个包补的就是这一层。

    #特性

    • 无损:原始 10 列全部保留在 Token 里,to_string 能逐字节还原(含多词 token、空节点、注释),不认识的列不会被"顺手规范化"。
    • 严格报错:parse 对列数、ID 形态、HEAD 数字性做硬校验,出错抛 ConlluError(line, message),行号直接给到源文件。
    • 规范校验:17 个 UPOS、37 个 UD v2 依存关系(含增强 ref)、ID 连续性、单根、自环、成环、DEPS 语法与范围,都按 CoNLL-U 规范检查。
    • 评测:evaluate(gold, pred) 给出 UAS / LAS(百分制),按 1-based 词序对齐,长度不等的句子计入 skipped_sentences 而不是静默丢掉。
    • 树查询:children / descendants / depth / is_ancestor / is_projective / roots。
    • 统计:句数、词数、多词 token、空节点、注释、最长句、平均句长、涉及的树库。
    • 对拍过:与 Python conllu 参考解析器在 UD_English-EWT 与 UD_Chinese-GSD 的 dev/test 四个切分上逐字段零差异,见 DIFF-REPORT.md。

    #安装

    moon add PaperY0/conllu

    #解析与序列化

    ///|
    test {
    let text = "1\tI\tI\tPRON\tPRP\t_\t2\tnsubj\t_\t_\n2\tsaw\tsee\tVERB\tVBD\t_\t0\troot\t_\t_\n3\tthe\tthe\tDET\tDT\t_\t4\tdet\t_\t_\n4\tcat\tcat\tNOUN\tNN\t_\t2\tobj\tSpaceAfter=No\t_\n5\t.\t.\tPUNCT\t.\t_\t2\tpunct\t_\t_\n"
    let sentences = @conllu.parse(text) catch { _ => [] }
    assert_eq(sentences.length(), 1)
    let sentence = sentences[0]
    assert_eq(sentence.word_count(), 5)
    assert_eq(sentence.words()[1].lemma, "see")
    // HEAD 是解析过的整数,可直接用于树操作
    assert_eq(sentence.words()[1].head(), Some(0))
    // 序列化补回规范要求的句末空行
    assert_eq(@conllu.to_string(sentences), text + "\n")
    }

    Sentence 同时保留注释和全部行:sentence.rows() 拿到 Word / Multiword / Empty 三类行,sentence.comment("text") 直接取 # text = ... 的值。

    #校验

    ///|
    test {
    let broken = "1\ta\ta\tNOUN\tNN\t_\t0\troot\t_\t_\n2\tb\tb\tNOUN\tNN\t_\t0\troot\t_\t_\n"
    let sentences = @conllu.parse(broken) catch { _ => [] }
    let diagnostics = @conllu.validate(sentences)
    assert_eq(@conllu.count_errors(diagnostics), 1)
    assert_eq(diagnostics[0].code, "ROOT_COUNT")
    }

    校验规则按 code 分类,方便 CI 里按类型放行或拦截:

    code含义
    ID_SEQUENCE词 ID 不是从 1 开始连续递增
    MULTIWORD_SPAN多词 token 的 5-6 越界
    HEAD_MISSING / HEAD_RANGE / HEAD_SELFHEAD 非数字、越界、指向自己
    ROOT_COUNT根节点不是恰好 1 个
    CYCLE沿 HEAD 走不到根
    DEPS_FORMAT / DEPS_RANGE / DEPS_DEPREL / DEPS_MISSINGDEPS 语法、范围、关系名、空节点缺关系
    UPOS_UNKNOWN / DEPREL_UNKNOWN不在 UD v2 标签集内
    FORM_EMPTY / FORM_SPACEFORM 为空或含空白

    #评测 UAS / LAS

    ///|
    test {
    let gold = "1\ta\ta\tNOUN\tNN\t_\t2\tnsubj\t_\t_\n2\tb\tb\tVERB\tVB\t_\t0\troot\t_\t_\n"
    let pred = "1\ta\ta\tNOUN\tNN\t_\t0\tnsubj\t_\t_\n2\tb\tb\tVERB\tVB\t_\t0\troot\t_\t_\n"
    let g = @conllu.parse(gold) catch { _ => [] }
    let p = @conllu.parse(pred) catch { _ => [] }
    let result = @conllu.evaluate(g, p)
    assert_eq(result.uas(), 50.0)
    assert_eq(result.las(), 50.0)
    }

    #命令行

    包自带 cmd/main,五个子命令都直接吃文件:

    moon run cmd/main -- stats treebank.conllu # 句数/词数/多词/空节点/最长句/树库 moon run cmd/main -- validate treebank.conllu # 诊断列表,有 error 时退出码 1 moon run cmd/main -- eval gold.conllu pred.conllu moon run cmd/main -- dump treebank.conllu # 归一化 dump,用于对拍 moon run cmd/main -- roundtrip treebank.conllu # 解析再序列化,校验无损

    dump 的格式是稳定的行式契约(D/S/w/m/e 前缀),差分脚本就靠它跟别的实现比对:

    D 2001 S 7 0 0 4 w 1 From ADP 3 case

    #差分对拍

    tools/diff_test.py 把本包的 dump 与 Python conllu 的输出逐行逐字段比对,另外检查 validate 零 error、roundtrip 无损、自评 UAS=LAS=100%。

    python3 tools/fetch_corpus.py # 下载 UD 官方语料到 data/ud/(语料不随包分发) python3 tools/diff_test.py --report

    当前结果(详见 DIFF-REPORT.md):

    文件句数dump 行数字段不一致
    en_ewt-ud-dev2001275130
    en_ewt-ud-test2077275280
    zh_gsd-ud-dev500131660
    zh_gsd-ud-test500125110

    对拍过程里发现并修掉的两个真实 bug:8.1:obj 这类指向空节点的 DEPS 被当成数字 81(越界误报),以及增强 DEPS 里合法的 ref 关系被判成未知标签。

    #许可与语料

    代码 Apache-2.0。UD 树库是 CC BY-SA 4.0,只作为测试夹具下载,不进仓库、不进发布包;引用请遵循 Universal Dependencies 项目的署名要求。

    ConlluError

    pub(all) suberror ConlluError {
    ConlluError(Int, String)
    }

    Raised when the input is not well-formed at the lexical level (wrong column count, unparsable ID, ...). Structural problems are reported by validate so that a broken tree can still be inspected.

    ConlluError::line

    fn ConlluError::line(self : ConlluError) -> Int

    The 1-based source line a ConlluError points at.

    ConlluError::message

    fn ConlluError::message(self : ConlluError) -> String

    Diagnostic

    pub struct Diagnostic {
    line : Int
    severity : Severity
    code : String
    message : String
    }

    One validation finding, tied to a physical line of the source file.

    Diagnostic::to_string

    fn Diagnostic::to_string(self : Diagnostic) -> String

    EvalResult

    pub struct EvalResult {
    sentence_pairs : Int
    skipped_sentences : Int
    gold_tokens : Int
    aligned_tokens : Int
    uas_correct : Int
    las_correct : Int
    }

    Aggregated attachment scores of a predicted parse against a gold parse.

    EvalResult::las

    fn EvalResult::las(self : EvalResult) -> Double

    Labelled attachment score in percent.

    EvalResult::to_string

    fn EvalResult::to_string(self : EvalResult) -> String

    One-line summary of the evaluation, as printed by the CLI.

    EvalResult::uas

    fn EvalResult::uas(self : EvalResult) -> Double

    Unlabelled attachment score in percent.

    RowKind

    pub(all) enum RowKind {
    Word
    Multiword
    Empty
    }

    CoNLL-U row kind. A sentence mixes word rows, multiword-token rows (1-2) and empty-node rows (5.1).

    Sentence

    pub struct Sentence {
    comments : Array[String]
    rows : Array[Token]
    line : Int
    }

    One CoNLL-U sentence: the comment block that precedes it and its rows in file order.

    Sentence::comment

    fn Sentence::comment(self : Sentence, key : StringView) -> String?

    Sentence::comments

    fn Sentence::comments(self : Sentence) -> Array[String]

    Sentence::rows

    fn Sentence::rows(self : Sentence) -> Array[Token]

    Sentence::to_string

    fn Sentence::to_string(self : Sentence) -> String

    Serialises one sentence back to CoNLL-U, without the trailing blank line.

    Sentence::word

    fn Sentence::word(self : Sentence, index : Int) -> Token?

    1-based lookup of a word row; None when out of range or not a word row.

    Sentence::word_count

    fn Sentence::word_count(self : Sentence) -> Int

    Sentence::words

    fn Sentence::words(self : Sentence) -> Array[Token]

    The word rows of a sentence, in ID order.

    Severity

    pub(all) enum Severity {
    Error
    Warning
    }

    Severity of a validation diagnostic.

    Stats

    pub struct Stats {
    sentences : Int
    words : Int
    multiwords : Int
    empty_nodes : Int
    comments : Int
    max_length : Int
    treebanks : Array[String]
    }

    Corpus-level statistics of a CoNLL-U document.

    Stats::average_length

    fn Stats::average_length(self : Stats) -> Double

    Stats::to_string

    fn Stats::to_string(self : Stats) -> String

    Token

    pub struct Token {
    id_raw : String
    form : String
    lemma : String
    upos : String
    xpos : String
    feats : String
    head_raw : String
    deprel : String
    deps : String
    misc : String
    kind : RowKind
    start : Int
    end : Int
    line : Int
    }

    One physical row of a CoNLL-U sentence: the ten tab-separated columns plus positional information derived by the parser.

    Token::head

    fn Token::head(self : Token) -> Int?

    The HEAD column as an integer, or None when the column is _ (legal only for empty nodes).

    Token::is_empty_node

    fn Token::is_empty_node(self : Token) -> Bool

    Token::is_multiword

    fn Token::is_multiword(self : Token) -> Bool

    Token::is_word

    fn Token::is_word(self : Token) -> Bool

    Token::to_line

    fn Token::to_line(self : Token) -> String

    Formats a single row back to one line.

    children

    fn children(sentence : Sentence, head : Int) -> Array[Int]

    Direct children of a word (by 1-based index); the root's children are returned for head == 0.

    count_errors

    fn count_errors(diagnostics : Array[Diagnostic]) -> Int

    depth

    fn depth(sentence : Sentence, index : Int) -> Int

    Depth of a word in its tree: root children have depth 1.

    descendants

    fn descendants(sentence : Sentence, ancestor : Int) -> Array[Int]

    All words whose HEAD chain passes through ancestor.

    evaluate

    fn evaluate(gold : Array[Sentence], pred : Array[Sentence]) -> EvalResult

    Compares two parses sentence by sentence. Sentences whose word counts differ cannot be aligned token by token and are counted as skipped instead of silently scored.

    is_ancestor

    fn is_ancestor(sentence : Sentence, ancestor : Int, node : Int) -> Bool

    True when following HEAD links from node reaches ancestor.

    is_projective

    fn is_projective(sentence : Sentence) -> Bool

    A sentence is projective when no two dependency arcs cross. An arc head -> dep is projective iff every word strictly between them is a descendant of head.

    max_depth

    fn max_depth(sentence : Sentence) -> Int

    Longest root-to-leaf distance in the sentence.

    parse

    fn parse(text : String) -> Array[Sentence] raise ConlluError

    Parses a complete CoNLL-U document. Blank lines separate sentences; a leading comment block belongs to the sentence that follows it.

    roots

    fn roots(sentence : Sentence) -> Array[Int]

    Root words of a sentence (HEAD == 0).

    stats

    fn stats(sentences : Array[Sentence]) -> Stats

    to_string

    fn to_string(sentences : Array[Sentence]) -> String

    Serialises a document with one blank line between sentences.

    validate

    fn validate(sentences : Array[Sentence]) -> Array[Diagnostic]

    Validates every sentence of a document.

    validate_sentence

    fn validate_sentence(sentence : Sentence) -> Array[Diagnostic]

    Checks the structure of one sentence: ID sequence, HEAD targets, single root, acyclicity, DEPS syntax, tag sets and token surface constraints. Lexical errors are already rejected by parse, so everything reported here keeps a sentence inspectable.