conllu

    CoNLL-U / Universal Dependencies toolchain: lossless parse and serialize, spec validation, tree queries, UAS/LAS evaluation, corpus statistics, document grouping, coverage analysis, per-document evaluation, JSON reports.

    conllu
    universal-dependencies
    nlp
    dependency-parsing
    treebank
    Download zip
    Author
    Version
    0.5.0
    License
    Apache-2.0
    Last updated
    14 hours ago
    Downloads
    25

    #PaperY0/conllu

    CI mooncakes license

    MoonBit 里的 CoNLL-U / Universal Dependencies 工具链:无损解析与序列化、规范校验、依存树查询、文档/段落分组、语料覆盖分析、UAS/LAS 评测、语料统计,外加一个可直接跑的 CLI(含 gzip 读与写)。

    CoNLL-U 是 UD 树库(Universal Dependencies 项目,200+ 语言树库)的唯一交换格式:每句 10 列制表符分隔,句子之间空行分隔,# 开头是注释,ID 可以是 12(普通词)、5-6(多词 token)、5.1(空节点)。生态里做分词、词性的包已经不少,但句法层(HEAD/DEPREL 树、投射性、UAS/LAS)此前是空白,这个包补的就是这一层。

    #特性

    • 无损:原始 10 列全部保留在 Token 里,to_string 能逐字节还原(含多词 token、空节点、注释),不认识的列不会被"顺手规范化"。
    • 严格报错:parse 对列数、ID 形态、HEAD 数字性做硬校验,出错抛 ConlluError(line, message),行号直接给到源文件。
    • 规范校验:17 个 UPOS、37 个 UD v2 依存关系(含增强 ref)、ID 连续性、单根、自环、成环、DEPS 语法与范围,都按 CoNLL-U 规范检查。
    • 跨句校验:validate_documents 补上句子级检查看不到的东西——缺 # sent_id、sent_id 重复、# newdoc id 重用、同一文档内段落 id 重用。
    • 增强依存可读:Token::enhanced_deps() 把 DEPS 拆成 EnhancedDep(head, rel),空节点头保持 5.1,不会被折成单词索引 5。
    • 评测:evaluate(gold, pred) 给出 UAS / LAS(百分制),按 1-based 词序对齐,长度不等的句子计入 skipped_sentences 而不是静默丢掉。
    • 树查询:children / descendants / depth / is_ancestor / is_projective / roots / path / is_single_rooted。
    • 文档分组:按 # newdoc / # newpar 还原 Document → Paragraph → Sentence 三层结构,没有边界声明的文件仍算一个匿名文档。
    • 覆盖分析:coverage 一次遍历统计 UPOS 与 DEPREL 命中率、规范外标签、投射性与树深、长句与空节点,CLI 里可直接当语料门槛用;merge_coverage 把 dev/test 合成一张表。
    • 按文档评测:evaluate_by_document 按 # newdoc 分组算 UAS / LAS,最差的文档排最前;预测漏掉整篇文档时显示为 0 对齐词,不会把分数悄悄摊平。
    • 统计:句数、词数、多词 token、空节点、注释、最长句、平均句长、涉及的树库;Stats::to_json() 与 Coverage::to_json() 供别的语言直接取数。
    • 对拍过:与 Python conllu 参考解析器在 UD_English-EWT、UD_Chinese-GSD、UD_Japanese-GSD 的 dev/test 六个切分(6,128 句、107,091 行)上逐字段零差异,见 DIFF-REPORT.md。
    • 输出有 golden 兜底:11 个 CLI 调用的完整 stdout 钉在 tests/golden/,CI 每次比对,改格式必现形。

    #安装

    moon add PaperY0/conllu

    #解析与序列化

    ///|
    test {
    let text = "1\tI\tI\tPRON\tPRP\t_\t2\tnsubj\t_\t_\n2\tsaw\tsee\tVERB\tVBD\t_\t0\troot\t_\t_\n3\tthe\tthe\tDET\tDT\t_\t4\tdet\t_\t_\n4\tcat\tcat\tNOUN\tNN\t_\t2\tobj\tSpaceAfter=No\t_\n5\t.\t.\tPUNCT\t.\t_\t2\tpunct\t_\t_\n"
    let sentences = @conllu.parse(text) catch { _ => [] }
    assert_eq(sentences.length(), 1)
    let sentence = sentences[0]
    assert_eq(sentence.word_count(), 5)
    assert_eq(sentence.words()[1].lemma, "see")
    // HEAD 是解析过的整数,可直接用于树操作
    assert_eq(sentence.words()[1].head(), Some(0))
    // 序列化补回规范要求的句末空行
    assert_eq(@conllu.to_string(sentences), text + "\n")
    }

    Sentence 同时保留注释和全部行:sentence.rows() 拿到 Word / Multiword / Empty 三类行,sentence.comment("text") 直接取 # text = ... 的值。

    增强依存(DEPS 列)也能直接取结构化结果,空节点引用保持在 5.1 这种形态:

    ///|
    test {
    let text = "1\twho\twho\tPRON\tWP\t_\t4\tnsubj\t4:nsubj|5.1:ref\t_\n"
    let sentences = @conllu.parse(text) catch { _ => [] }
    let deps = sentences[0].word(1).unwrap().enhanced_deps()
    assert_eq(deps.length(), 2)
    assert_eq(deps[0].head(), "4")
    assert_eq(deps[1].head(), "5.1")
    assert_eq(deps[1].rel(), "ref")
    }

    #校验

    ///|
    test {
    let broken = "1\ta\ta\tNOUN\tNN\t_\t0\troot\t_\t_\n2\tb\tb\tNOUN\tNN\t_\t0\troot\t_\t_\n"
    let sentences = @conllu.parse(broken) catch { _ => [] }
    let diagnostics = @conllu.validate(sentences)
    assert_eq(@conllu.count_errors(diagnostics), 1)
    assert_eq(diagnostics[0].code, "ROOT_COUNT")
    }

    校验规则按 code 分类,方便 CI 里按类型放行或拦截:

    code含义
    ID_SEQUENCE词 ID 不是从 1 开始连续递增
    MULTIWORD_SPAN多词 token 的 5-6 越界
    HEAD_MISSING / HEAD_RANGE / HEAD_SELFHEAD 非数字、越界、指向自己
    ROOT_COUNT根节点不是恰好 1 个
    CYCLE沿 HEAD 走不到根
    DEPS_FORMAT / DEPS_RANGE / DEPS_DEPREL / DEPS_MISSINGDEPS 语法、范围、关系名、空节点缺关系
    UPOS_UNKNOWN / DEPREL_UNKNOWN不在 UD v2 标签集内
    FORM_EMPTY / FORM_SPACEFORM 为空或含空白
    SENT_ID_MISSING / SENT_ID_DUPLICATE缺 # sent_id(警告)/ 重复(错误),由 validate_documents 报
    NEWDOC_ID_DUPLICATE / NEWPAR_ID_DUPLICATE文档 id 重用 / 同一文档内段落 id 重用(警告)
    NEWPAR_WITHOUT_DOC# newpar 出现在任何 # newdoc 之前(警告)

    单句层面的错误 validate 就能报,但 id 重复这类问题只有把整份文件放在一起看才成立:

    ///|
    test {
    let text = "# newdoc id = d1\n# sent_id = s1\n1\ta\ta\tNOUN\tNN\t_\t0\troot\t_\t_\n\n# sent_id = s1\n1\tb\tb\tNOUN\tNN\t_\t0\troot\t_\t_\n"
    let sentences = @conllu.parse(text) catch { _ => [] }
    let diagnostics = @conllu.validate_documents(sentences)
    assert_eq(diagnostics.length(), 1)
    assert_eq(diagnostics[0].code, "SENT_ID_DUPLICATE")
    assert_eq(@conllu.count_errors(diagnostics), 1)
    }

    #文档与段落

    树库不是扁平的句子列表:# newdoc 与 # newpar 行标出每句属于哪个文档、哪一段。documents() 把这层关系还原成三层结构,id 直接取自注释;没有边界声明的文件返回一个 id 为空的匿名文档,而不是空数组。

    ///|
    test {
    let text = "# newdoc id = d1\n# newpar id = p1\n# sent_id = 1\n1\ta\ta\tNOUN\tNN\t_\t0\troot\t_\t_\n\n# newpar id = p2\n# sent_id = 2\n1\tb\tb\tNOUN\tNN\t_\t0\troot\t_\t_\n"
    let docs = @conllu.documents(@conllu.parse(text) catch { _ => [] })
    assert_eq(docs.length(), 1)
    assert_eq(docs[0].id(), "d1")
    assert_eq(docs[0].paragraph_count(), 2)
    assert_eq(docs[0].paragraphs()[0].id(), "p1")
    assert_eq(docs[0].paragraphs()[1].id(), "p2")
    assert_eq(docs[0].sentence_count(), 2)
    }

    # newdoc 一行同时开启新段落,因此文档不会继承上一段的尾巴;分组信息本身就是普通注释,Document::to_string 写回时这些行原样保留。

    #语料覆盖分析

    句数词数只能说明体量,说明不了质量。coverage 一次遍历回答四个问题:规范里的 17 个 UPOS、37 个 DEPREL 各用到了几个;有没有规范外的标签;树有多深、多少句不投射;长句和空节点有多少。

    ///|
    test {
    let text = "1\ta\ta\tNOUN\tNN\t_\t0\troot\t_\t_\n2\tb\tb\tUNKNOWN\tNN\t_\t1\tweird\t_\t_\n"
    let cov = @conllu.coverage(@conllu.parse(text) catch { _ => [] })
    assert_eq(cov.upos_used.length(), 1)
    assert_eq(cov.unknown_upos[0], "UNKNOWN")
    assert_eq(cov.unknown_deprel[0], "weird")
    assert_false(cov.full_upos())
    assert_eq(cov.projectivity(), 100.0)
    }

    多文件时用 merge_coverage 或直接给 CLI 多个路径:标签取并集、计数相加、缺失标签按规范重算,平均树深按词数加权重算,不会出现"平均的平均":

    moon run cmd/main -- analyze data/ud/en_ewt-ud-dev.conllu data/ud/zh_gsd-ud-test.conllu

    file sentences words upos deprel unknown projective depth en_ewt-ud-dev.conllu 2001 25148 17/17 36/37 0 98.35% 11 zh_gsd-ud-test.conllu 500 12010 16/17 29/37 0 97.2% 10 TOTAL 2501 37158 17/17 37/37 0 98.12% 11

    #评测 UAS / LAS

    ///|
    test {
    let gold = "1\ta\ta\tNOUN\tNN\t_\t2\tnsubj\t_\t_\n2\tb\tb\tVERB\tVB\t_\t0\troot\t_\t_\n"
    let pred = "1\ta\ta\tNOUN\tNN\t_\t0\tnsubj\t_\t_\n2\tb\tb\tVERB\tVB\t_\t0\troot\t_\t_\n"
    let g = @conllu.parse(gold) catch { _ => [] }
    let p = @conllu.parse(pred) catch { _ => [] }
    let result = @conllu.evaluate(g, p)
    assert_eq(result.uas(), 50.0)
    assert_eq(result.las(), 50.0)
    }

    整体分数会掩盖"某几篇文档全错"的情况,evaluate_by_document 按 gold 的 # newdoc 分组分别算分,最差的排最前:

    ///|
    test {
    let gold = "# newdoc id = d1\n1\ta\ta\tNOUN\tNN\t_\t2\tnsubj\t_\t_\n2\tb\tb\tVERB\tVB\t_\t0\troot\t_\t_\n"
    let pred = "# newdoc id = d1\n1\ta\ta\tNOUN\tNN\t_\t0\troot\t_\t_\n2\tb\tb\tVERB\tVB\t_\t1\tnsubj\t_\t_\n"
    let scores = @conllu.evaluate_by_document(
    @conllu.parse(gold) catch {
    _ => []
    },
    @conllu.parse(pred) catch {
    _ => []
    },
    )
    assert_eq(scores.length(), 1)
    assert_eq(scores[0].id(), "d1")
    assert_eq(scores[0].sentences(), 1)
    assert_eq(scores[0].tokens(), 2)
    // 两条边都接反了:UAS 与 LAS 都是 0。
    assert_eq(scores[0].las(), 0.0)
    }

    CLI 里 eval <gold> <pred> --by-document 直接打表,末尾附整体 total。

    #命令行

    包自带 cmd/main,子命令都直接吃文件,纯文本和 .conllu.gz 都支持,输出去 .gz 时自动压缩:

    moon run cmd/main -- stats treebank.conllu # 句数/词数/多词/空节点/最长句/树库 moon run cmd/main -- stats treebank.conllu --format json moon run cmd/main -- validate treebank.conllu # 诊断列表,有 error 时退出码 1 moon run cmd/main -- validate treebank.conllu --documents # 加上跨句 id 检查 moon run cmd/main -- validate treebank.conllu --format json # 每条诊断一个对象,退出码不变 moon run cmd/main -- eval gold.conllu pred.conllu moon run cmd/main -- eval gold.conllu pred.conllu --by-document moon run cmd/main -- labels gold.conllu pred.conllu # 按关系拆开的 LAS moon run cmd/main -- labels gold.conllu pred.conllu --format csv moon run cmd/main -- docs treebank.conllu # 文档/段落结构与句数 moon run cmd/main -- analyze treebank.conllu # 覆盖报告,规范外标签时退出码 1 moon run cmd/main -- analyze dev.conllu test.conllu # 多文件表格 + TOTAL moon run cmd/main -- analyze dev.conllu test.conllu --format json moon run cmd/main -- dump treebank.conllu # 归一化 dump,用于对拍 moon run cmd/main -- dump treebank.conllu --format json moon run cmd/main -- dump treebank.conllu --format tsv moon run cmd/main -- roundtrip treebank.conllu # 解析再序列化,校验无损 moon run cmd/main -- compress treebank.conllu out.conllu.gz # 规范序列化 + gzip 写回

    compress 会把自己写出的文件再读回来比对,写坏了不会静默产出一个"看着像"的文件。UD_English-EWT dev 实测 1,899,904 → 357,823 字节(18.83%)。

    validate --format json 输出 {"file": ..., "diagnostics": [...], "errors": N, "warnings": M},每条诊断是 {"line": 7, "severity": "error", "code": "ROOT_COUNT", "message": "..."}。CI 里想按 code 过滤或统计,不用再正则解析文本行;退出码规则与文本模式一致(有 error 就是 1)。

    moon run cmd/main -- validate corpus.conllu --format json | jq -r '.diagnostics[].code' | sort | uniq -c

    labels --format csv 输出 label,gold,correct,las 四列,按 RFC 4180 处理引号,可直接喂给表格或 pandas;默认仍是终端对齐表。

    dump 的格式是稳定的行式契约(D/S/w/m/e 前缀),差分脚本就靠它跟别的实现比对:

    D 2001 S 7 0 0 4 w 1 From ADP 3 case

    需要结构化输出时用 --format json:一份文件一个 JSON 对象,句子带行号、注释、词/多词/空节点计数,token 保留原始 10 列值,方便别的语言写包装器。

    要直接进表格就用 --format tsv:固定 13 列表头(sentence / sent_line / kind / id / form / lemma / upos / xpos / feats / head / deprel / deps / misc),一行一个 token。CoNLL-U 的列本身不允许制表符,所以不用处理引号:

    import pandas as pd df = pd.read_csv("dump.tsv", sep="\t", dtype=str) df[df.kind == "word"].groupby("deprel").size().sort_values(ascending=False)

    #差分对拍

    tools/diff_test.py 把本包的 dump 与 Python conllu 的输出逐行逐字段比对,另外检查 validate 零 error、roundtrip 无损、自评 UAS=LAS=100%。

    python3 tools/fetch_corpus.py # 下载 UD 官方语料到 data/ud/(语料不随包分发) python3 tools/diff_test.py --report

    当前结果(详见 DIFF-REPORT.md):

    文件句数dump 行数字段不一致
    en_ewt-ud-dev2001275130
    en_ewt-ud-test2077275280
    zh_gsd-ud-dev500131660
    zh_gsd-ud-test500125110
    ja_gsd-ud-dev507127950
    ja_gsd-ud-test543135780

    对拍过程里发现并修掉的两个真实 bug:8.1:obj 这类指向空节点的 DEPS 被当成数字 81(越界误报),以及增强 DEPS 里合法的 ref 关系被判成未知标签。

    golden 回归。对拍依赖外部语料,只能证明"数字对",证明不了"输出长什么样"。tests/golden/cases.json 把十一个 CLI 调用(stats / validate / dump 三种格式 / eval / docs / analyze / 跨句校验 / JSON 诊断)连同各自的期望 stdout 一起钉住,列顺序、标签大小写、JSON 键名、pad 宽度一改就会被抓出来:

    python3 tools/golden_test.py # 比对,11/11 必须全过 python3 tools/golden_test.py --update # 有意改动输出后重新生成

    golden 集跟着 moon test 一起进 CI:integration job 在差分对拍后跑一遍,所以"本地测试绿了但 CLI 文本悄悄变了"这种情况不会再漏。

    #API 参考

    完整文档注释可用 moon doc --serve 在本地预览(默认 http://127.0.0.1:3000),发布后也会出现在 mooncakes 的文档页。

    解析与序列化

    函数说明
    parse(text) -> Array[Sentence] raise ConlluError按空行分句,校验 10 列与 ID 形态,出错带行号
    to_string(sentences) -> String序列化整份文档,每句后补规范空行
    Sentence::to_string()单句序列化
    Sentence::to_json()单句 JSON 对象(计数、注释、逐 token 原始列)
    to_json(sentences, source?=...)整份文档 JSON,{"source": ..., "sentences": [...]}
    Sentence::words() / rows() / comments()词行 / 全部行 / 注释
    Sentence::word_count()词行数量(不含多词与空节点)
    Sentence::word(i) -> Token?按 1-based 词序取词
    Sentence::comment(key)取 # key = value 的 value
    Sentence::comment_value(key)同上的通用版本,Some(value) / None
    Token::head() -> Int32?HEAD 解析为整数,非数字返回 None
    Token::is_word() / is_multiword() / is_empty_node()行类型判定
    Token::to_line()还原一行 10 列
    Token::enhanced_deps() -> Array[EnhancedDep]DEPS 拆成 head:rel,空节点头原样保留
    EnhancedDep::head() / rel() / to_string()单条增强依存

    校验

    函数说明
    validate(sentences) -> Array[Diagnostic]全文档校验
    validate_sentence(sentence)单句校验
    validate_documents(sentences)跨句 checks:sent_id / newdoc / newpar 的唯一性与缺失
    Diagnostic::to_string()line 7: error [CODE] message
    Diagnostic::to_json()单条诊断的 JSON 对象
    diagnostics_to_json(diagnostics)诊断列表作为 JSON 数组
    count_errors(diagnostics) -> Int32只数 Error 级别
    Diagnostic::to_string()line 12: error [ROOT_COUNT] ...

    依存树

    函数说明
    children(s, head) / descendants(s, node)直接孩子 / 全部后代
    path(s, from, to) -> Array[Int32]?从 from 沿 HEAD 到祖先 to(to = 0 走到根)
    is_ancestor(s, a, b)a 是否为 b 的祖先
    depth(s, i) / max_depth(s)深度 / 最大深度
    is_projective(s)是否无交叉弧
    roots(s) / is_single_rooted(s)根节点 / 是否单根

    文档与段落

    函数说明
    documents(sentences) -> Array[Document]按 # newdoc / # newpar 分组
    Document::id() / paragraph_count() / sentence_count()文档 id 与两级计数
    Document::paragraphs() / sentences()段落列表 / 展开成句子列表
    Paragraph::id() / sentences() / sentence_count()段落 id 与句数
    Sentence::has_newdoc() / has_newpar()是否开启新文档 / 新段落
    Sentence::newdoc_id() / newpar_id()对应注释的 id 值

    覆盖分析

    函数说明
    coverage(sentences) -> Coverage标签命中、规范外标签、树形与句长统计
    merge_coverage(covs) -> Coverage合并多文件:标签取并集、计数相加、平均树深按词数重算
    Coverage::upos_used() / deprel_used()实际出现的标签(字母序)
    Coverage::upos_missing() / deprel_missing()规范里没出现的标签
    Coverage::projectivity()投射句占比(百分制)
    Coverage::full_upos() / full_deprel()是否覆盖全部规范标签
    Coverage::to_string()多行报告,analyze 直接打印
    Coverage::to_json()同一份数据作为 JSON 对象

    评测与统计

    函数说明
    evaluate(gold, pred) -> EvalResult对齐后计算 UAS / UUAS / LAS
    EvalResult::uas() / uuas() / las()百分制分数;uuas 不计方向
    evaluate_by_label(gold, pred)按 gold 关系拆开的 LAS,最差的排最前
    evaluate_by_document(gold, pred)按 # newdoc 分组评测,最差的文档排最前
    DocumentScore::id() / sentences() / tokens() / uas() / las()单文档分数
    LabelScore::las() / to_string()单关系准确率
    stats(sentences) -> Stats句数、词数、多词、空节点、注释、最长句、树库
    Stats::average_length() / to_string()平均句长 / 一行摘要
    Stats::to_json()统计结果作为 JSON 对象

    #端到端

    一份样例(examples/sample.conllu,随仓库提供)把全部子命令走一遍,输出都是真实跑出来的:

    moon run cmd/main -- stats examples/sample.conllu # sentences=1 words=5 multiword=0 empty=0 comments=2 max_len=5 avg_len=5 docs=0 # errors=0 moon run cmd/main -- validate examples/sample.conllu --documents # diagnostics=0 errors=0 warnings=0 moon run cmd/main -- docs examples/sample.conllu # doc 1 id=- sentences= 1 paragraphs=1 # documents=1 paragraphs=1 moon run cmd/main -- dump examples/sample.conllu --format tsv # sentence sent_line kind id form lemma upos xpos feats head deprel deps misc # 1 1 word 1 I I PRON PRP _ 2 nsubj _ _ moon run cmd/main -- compress examples/sample.conllu /tmp/sample.conllu.gz # compress: 1 sentences 216 -> 186 bytes (86.11% of input) # verify: /tmp/sample.conllu.gz reads back identically

    评测流程则是三步:stats / validate 确认语料可用,eval 出总分,eval --by-document 与 labels 定位错在哪些文档、哪些关系:

    moon run cmd/main -- eval gold.conllu pred.conllu moon run cmd/main -- eval gold.conllu pred.conllu --by-document moon run cmd/main -- labels gold.conllu pred.conllu --format csv

    #许可与语料

    代码 Apache-2.0。UD 树库是 CC BY-SA 4.0,只作为测试夹具下载,不进仓库、不进发布包;引用请遵循 Universal Dependencies 项目的署名要求。

    ConlluError

    pub(all) suberror ConlluError {
    ConlluError(Int, String)
    }

    Raised when the input is not well-formed at the lexical level (wrong column count, unparsable ID, ...). Structural problems are reported by validate so that a broken tree can still be inspected.

    ConlluError::line

    fn ConlluError::line(self : ConlluError) -> Int

    The 1-based source line a ConlluError points at.

    ConlluError::message

    fn ConlluError::message(self : ConlluError) -> String

    Coverage

    pub struct Coverage {
    sentences : Int
    words : Int
    upos_used : Array[String]
    upos_missing : Array[String]
    deprel_used : Array[String]
    deprel_missing : Array[String]
    unknown_upos : Array[String]
    unknown_deprel : Array[String]
    untagged : Int
    projective : Int
    non_projective : Int
    single_rooted : Int
    multi_rooted : Int
    max_depth : Int
    avg_depth : Double
    depth_total : Int
    long_sentences : Int
    empty_nodes : Int
    multiword_tokens : Int
    documents : Int
    paragraphs : Int
    }

    What a corpus actually covers: which universal tags and relations appear, which tags it uses that the UD spec does not define, how well-formed its trees are, and the shape of its sentences. Answers the questions a corpus maintainer asks before trusting a treebank.

    Coverage::full_deprel

    fn Coverage::full_deprel(self : Coverage) -> Bool

    Same for the 37 universal relations.

    Coverage::full_upos

    fn Coverage::full_upos(self : Coverage) -> Bool

    True when every universal tag and relation of the spec appears at least once. A false here is not a defect, but it does tell you the corpus will not exercise those labels in a parser evaluation.

    Coverage::projectivity

    fn Coverage::projectivity(self : Coverage) -> Double

    Share of sentences whose tree is projective, in percent.

    Coverage::to_json

    fn Coverage::to_json(self : Coverage) -> String

    The same report as one JSON object, for callers that build dashboards or diff two corpora programmatically. Label arrays are plain string arrays.

    Coverage::to_string

    fn Coverage::to_string(self : Coverage) -> String

    Human-readable report, one section per dimension. Printed by conllu analyze.

    Diagnostic

    pub struct Diagnostic {
    line : Int
    severity : Severity
    code : String
    message : String
    }

    One validation finding, tied to a physical line of the source file.

    Diagnostic::to_json

    fn Diagnostic::to_json(self : Diagnostic) -> String

    One diagnostic as a JSON object, for CI steps that need to filter by code instead of regexing line 12: error [ROOT_COUNT] ....

    Diagnostic::to_string

    fn Diagnostic::to_string(self : Diagnostic) -> String

    Document

    pub struct Document {
    id : String
    paragraphs : Array[Paragraph]
    }

    A document (# newdoc) split into the paragraphs it contains. Sentences that appear before the first # newdoc line form one document with an empty id, which is what a file without document boundaries looks like.

    Document::id

    fn Document::id(self : Document) -> String

    Document::paragraph_count

    fn Document::paragraph_count(self : Document) -> Int

    Document::paragraphs

    fn Document::paragraphs(self : Document) -> Array[Paragraph]

    Document::sentence_count

    fn Document::sentence_count(self : Document) -> Int

    Document::sentences

    fn Document::sentences(self : Document) -> Array[Sentence]

    All sentences of the document, in file order.

    Document::to_string

    fn Document::to_string(self : Document) -> String

    Serialises the document with the canonical writer, one blank line between sentences. The # newdoc / # newpar lines travel as ordinary comments, so the grouping survives a write.

    DocumentScore

    pub struct DocumentScore {
    id : String
    result : EvalResult
    }

    Scores of one document: the aggregate EvalResult of its sentences, keyed by the # newdoc id of the gold file. A parser can be 90% LAS overall and still lose entire documents, which is what this view exposes.

    DocumentScore::id

    fn DocumentScore::id(self : DocumentScore) -> String

    DocumentScore::las

    fn DocumentScore::las(self : DocumentScore) -> Double

    DocumentScore::sentences

    fn DocumentScore::sentences(self : DocumentScore) -> Int

    DocumentScore::to_string

    fn DocumentScore::to_string(self : DocumentScore) -> String

    DocumentScore::tokens

    fn DocumentScore::tokens(self : DocumentScore) -> Int

    DocumentScore::uas

    fn DocumentScore::uas(self : DocumentScore) -> Double

    EnhancedDep

    pub struct EnhancedDep {
    head : String
    rel : String
    }

    One enhanced dependency of a word: the head exactly as written in the DEPS column (a word index or an empty-node index such as 5.1) and its relation.

    EnhancedDep::head

    fn EnhancedDep::head(self : EnhancedDep) -> String

    EnhancedDep::rel

    fn EnhancedDep::rel(self : EnhancedDep) -> String

    EnhancedDep::to_string

    fn EnhancedDep::to_string(self : EnhancedDep) -> String

    EvalResult

    pub struct EvalResult {
    sentence_pairs : Int
    skipped_sentences : Int
    gold_tokens : Int
    aligned_tokens : Int
    uas_correct : Int
    uuas_correct : Int
    las_correct : Int
    }

    Aggregated attachment scores of a predicted parse against a gold parse.

    EvalResult::las

    fn EvalResult::las(self : EvalResult) -> Double

    Labelled attachment score in percent.

    EvalResult::to_string

    fn EvalResult::to_string(self : EvalResult) -> String

    One-line summary of the evaluation, as printed by the CLI.

    EvalResult::uas

    fn EvalResult::uas(self : EvalResult) -> Double

    Unlabelled attachment score in percent.

    EvalResult::uuas

    fn EvalResult::uuas(self : EvalResult) -> Double

    Undirected attachment score in percent: a token counts as correct when it points at the same head as the gold parse, ignoring the direction of the dependency. Useful when a parser gets the attachment right but the direction or the label wrong.

    LabelScore

    pub struct LabelScore {
    label : String
    gold : Int
    correct : Int
    }

    Labelled accuracy of one dependency relation.

    LabelScore::las

    fn LabelScore::las(self : LabelScore) -> Double

    Labelled accuracy of this relation in percent.

    LabelScore::to_string

    fn LabelScore::to_string(self : LabelScore) -> String

    One-line summary: label=nsubj gold=210 correct=198 las=94.29%.

    Paragraph

    pub struct Paragraph {
    id : String
    sentences : Array[Sentence]
    }

    A run of sentences inside one paragraph. id is the # newpar id value, or the empty string when the corpus does not name paragraphs.

    Paragraph::id

    fn Paragraph::id(self : Paragraph) -> String

    Paragraph::sentence_count

    fn Paragraph::sentence_count(self : Paragraph) -> Int

    Paragraph::sentences

    fn Paragraph::sentences(self : Paragraph) -> Array[Sentence]

    RowKind

    pub(all) enum RowKind {
    Word
    Multiword
    Empty
    }

    CoNLL-U row kind. A sentence mixes word rows, multiword-token rows (1-2) and empty-node rows (5.1).

    Sentence

    pub struct Sentence {
    comments : Array[String]
    rows : Array[Token]
    line : Int
    }

    One CoNLL-U sentence: the comment block that precedes it and its rows in file order.

    Sentence::comment

    fn Sentence::comment(self : Sentence, key : StringView) -> String?

    Sentence::comment_value

    fn Sentence::comment_value(self : Sentence, key : StringView) -> String?

    Value of a sentence comment written as # <key> = <value>, the form used by # newdoc id = ... and # newpar id = .... Returns None when the sentence carries no such comment.

    Sentence::comments

    fn Sentence::comments(self : Sentence) -> Array[String]

    Sentence::has_newdoc

    fn Sentence::has_newdoc(self : Sentence) -> Bool

    True when the sentence opens a new document (# newdoc).

    Sentence::has_newpar

    fn Sentence::has_newpar(self : Sentence) -> Bool

    True when the sentence opens a new paragraph (# newpar).

    Sentence::newdoc_id

    fn Sentence::newdoc_id(self : Sentence) -> String?

    The document id of a # newdoc id = ... line, if any.

    Sentence::newpar_id

    fn Sentence::newpar_id(self : Sentence) -> String?

    The paragraph id of a # newpar id = ... line, if any.

    Sentence::rows

    fn Sentence::rows(self : Sentence) -> Array[Token]

    Sentence::to_json

    fn Sentence::to_json(self : Sentence) -> String

    One sentence as a JSON object: line, row counts, comments and the tokens.

    Sentence::to_string

    fn Sentence::to_string(self : Sentence) -> String

    Serialises one sentence back to CoNLL-U, without the trailing blank line.

    Sentence::word

    fn Sentence::word(self : Sentence, index : Int) -> Token?

    1-based lookup of a word row; None when out of range or not a word row.

    Sentence::word_count

    fn Sentence::word_count(self : Sentence) -> Int

    Sentence::words

    fn Sentence::words(self : Sentence) -> Array[Token]

    The word rows of a sentence, in ID order.

    Severity

    pub(all) enum Severity {
    Error
    Warning
    }

    Severity of a validation diagnostic.

    Stats

    pub struct Stats {
    sentences : Int
    words : Int
    multiwords : Int
    empty_nodes : Int
    comments : Int
    max_length : Int
    treebanks : Array[String]
    }

    Corpus-level statistics of a CoNLL-U document.

    Stats::average_length

    fn Stats::average_length(self : Stats) -> Double

    Stats::to_json

    fn Stats::to_json(self : Stats) -> String

    Same numbers as Stats::to_string, as one JSON object, so a wrapper in another language does not have to parse the sentence-level summary.

    Stats::to_string

    fn Stats::to_string(self : Stats) -> String

    Token

    pub struct Token {
    id_raw : String
    form : String
    lemma : String
    upos : String
    xpos : String
    feats : String
    head_raw : String
    deprel : String
    deps : String
    misc : String
    kind : RowKind
    start : Int
    end : Int
    line : Int
    }

    One physical row of a CoNLL-U sentence: the ten tab-separated columns plus positional information derived by the parser.

    Token::enhanced_deps

    fn Token::enhanced_deps(self : Token) -> Array[EnhancedDep]

    Structured form of the DEPS column: 1:nsubj|5.1:ref becomes two entries, with the head kept as a string so an empty-node reference is not flattened into a plain word index. Returns an empty array for _.

    Token::head

    fn Token::head(self : Token) -> Int?

    The HEAD column as an integer, or None when the column is _ (legal only for empty nodes).

    Token::is_empty_node

    fn Token::is_empty_node(self : Token) -> Bool

    Token::is_multiword

    fn Token::is_multiword(self : Token) -> Bool

    Token::is_word

    fn Token::is_word(self : Token) -> Bool

    Token::to_line

    fn Token::to_line(self : Token) -> String

    Formats a single row back to one line.

    children

    fn children(sentence : Sentence, head : Int) -> Array[Int]

    Direct children of a word (by 1-based index); the root's children are returned for head == 0.

    count_errors

    fn count_errors(diagnostics : Array[Diagnostic]) -> Int

    coverage

    fn coverage(sentences : Array[Sentence]) -> Coverage

    Measures one corpus.

    depth

    fn depth(sentence : Sentence, index : Int) -> Int

    Depth of a word in its tree: root children have depth 1.

    descendants

    fn descendants(sentence : Sentence, ancestor : Int) -> Array[Int]

    All words whose HEAD chain passes through ancestor.

    diagnostics_to_json

    fn diagnostics_to_json(diagnostics : Array[Diagnostic]) -> String

    A diagnostic list as a JSON array.

    documents

    fn documents(sentences : Array[Sentence]) -> Array[Document]

    Groups sentences by # newdoc and then by # newpar, in file order. A # newdoc line always starts a new paragraph as well, so a document never inherits the tail of the previous one.

    evaluate

    fn evaluate(gold : Array[Sentence], pred : Array[Sentence]) -> EvalResult

    Compares two parses sentence by sentence. Sentences whose word counts differ cannot be aligned token by token and are counted as skipped instead of silently scored.

    evaluate_by_document

    fn evaluate_by_document(gold : Array[Sentence], pred : Array[Sentence]) -> Array[DocumentScore]

    Evaluates document by document instead of sentence by sentence. Documents come from # newdoc in the gold file; predictions are matched by position, so a prediction that merged or dropped a document shows up as a document with skipped sentences rather than as a silently reshuffled score. Documents are returned worst-first by LAS.

    evaluate_by_label

    fn evaluate_by_label(gold : Array[Sentence], pred : Array[Sentence]) -> Array[LabelScore]

    Labelled accuracy broken down by the gold relation, worst first, so a parser can see which relations it confuses instead of a single aggregate number.

    is_ancestor

    fn is_ancestor(sentence : Sentence, ancestor : Int, node : Int) -> Bool

    True when following HEAD links from node reaches ancestor.

    is_projective

    fn is_projective(sentence : Sentence) -> Bool

    A sentence is projective when no two dependency arcs cross. An arc head -> dep is projective iff every word strictly between them is a descendant of head.

    is_single_rooted

    fn is_single_rooted(sentence : Sentence) -> Bool

    True when the sentence has exactly one word attached to the root. A well-formed CoNLL-U sentence is single-rooted by definition; this is the cheap check for tooling that only needs that one property.

    max_depth

    fn max_depth(sentence : Sentence) -> Int

    Longest root-to-leaf distance in the sentence.

    merge_coverage

    fn merge_coverage(items : Array[Coverage]) -> Coverage

    Combines per-file coverages into one corpus-wide view: counters add up, used labels are unioned, missing labels are recomputed against the spec, and the average depth is rebuilt from the token-weighted depth total so a merged report does not average averages.

    parse

    fn parse(text : String) -> Array[Sentence] raise ConlluError

    Parses a complete CoNLL-U document. Blank lines separate sentences; a leading comment block belongs to the sentence that follows it.

    path

    fn path(sentence : Sentence, from : Int, to : Int) -> Array[Int]?

    Dependency path from from up to to, both inclusive, in the order from, ..., to. Returns None when to is not an ancestor of from or an index is out of range. to = 0 walks all the way up to the artificial root.

    roots

    fn roots(sentence : Sentence) -> Array[Int]

    Root words of a sentence (HEAD == 0).

    stats

    fn stats(sentences : Array[Sentence]) -> Stats

    to_json

    fn to_json(sentences : Array[Sentence], source? : String) -> String

    Whole document as one JSON object: {"source": ..., "sentences": [...]}. source is omitted when no path is passed. Used by dump --format json and by wrappers that would otherwise re-split the line-oriented dump.

    to_string

    fn to_string(sentences : Array[Sentence]) -> String

    Serialises a document with one blank line between sentences.

    validate

    fn validate(sentences : Array[Sentence]) -> Array[Diagnostic]

    Validates every sentence of a document.

    validate_documents

    fn validate_documents(sentences : Array[Sentence]) -> Array[Diagnostic]

    Checks the properties that only make sense across sentences: every sentence carrying a # sent_id, no repeated # sent_id, and no document or paragraph id reused inside the same scope. Sentence-level structure is already covered by validate_sentence; these findings are about the file as a corpus.

    validate_sentence

    fn validate_sentence(sentence : Sentence) -> Array[Diagnostic]

    Checks the structure of one sentence: ID sequence, HEAD targets, single root, acyclicity, DEPS syntax, tag sets and token surface constraints. Lexical errors are already rejected by parse, so everything reported here keeps a sentence inspectable.