CoNLL-U / Universal Dependencies toolchain: lossless parse and serialize, spec validation, tree queries, UAS/LAS evaluation, corpus statistics.
moon add PaperY0/conllu///|
test {
let text = "1\tI\tI\tPRON\tPRP\t_\t2\tnsubj\t_\t_\n2\tsaw\tsee\tVERB\tVBD\t_\t0\troot\t_\t_\n3\tthe\tthe\tDET\tDT\t_\t4\tdet\t_\t_\n4\tcat\tcat\tNOUN\tNN\t_\t2\tobj\tSpaceAfter=No\t_\n5\t.\t.\tPUNCT\t.\t_\t2\tpunct\t_\t_\n"
let sentences = @conllu.parse(text) catch { _ => [] }
assert_eq(sentences.length(), 1)
let sentence = sentences[0]
assert_eq(sentence.word_count(), 5)
assert_eq(sentence.words()[1].lemma, "see")
// HEAD 是解析过的整数,可直接用于树操作
assert_eq(sentence.words()[1].head(), Some(0))
// 序列化补回规范要求的句末空行
assert_eq(@conllu.to_string(sentences), text + "\n")
}///|
test {
let broken = "1\ta\ta\tNOUN\tNN\t_\t0\troot\t_\t_\n2\tb\tb\tNOUN\tNN\t_\t0\troot\t_\t_\n"
let sentences = @conllu.parse(broken) catch { _ => [] }
let diagnostics = @conllu.validate(sentences)
assert_eq(@conllu.count_errors(diagnostics), 1)
assert_eq(diagnostics[0].code, "ROOT_COUNT")
}| code | 含义 |
|---|---|
| ID_SEQUENCE | 词 ID 不是从 1 开始连续递增 |
| MULTIWORD_SPAN | 多词 token 的 5-6 越界 |
| HEAD_MISSING / HEAD_RANGE / HEAD_SELF | HEAD 非数字、越界、指向自己 |
| ROOT_COUNT | 根节点不是恰好 1 个 |
| CYCLE | 沿 HEAD 走不到根 |
| DEPS_FORMAT / DEPS_RANGE / DEPS_DEPREL / DEPS_MISSING | DEPS 语法、范围、关系名、空节点缺关系 |
| UPOS_UNKNOWN / DEPREL_UNKNOWN | 不在 UD v2 标签集内 |
| FORM_EMPTY / FORM_SPACE | FORM 为空或含空白 |
///|
test {
let gold = "1\ta\ta\tNOUN\tNN\t_\t2\tnsubj\t_\t_\n2\tb\tb\tVERB\tVB\t_\t0\troot\t_\t_\n"
let pred = "1\ta\ta\tNOUN\tNN\t_\t0\tnsubj\t_\t_\n2\tb\tb\tVERB\tVB\t_\t0\troot\t_\t_\n"
let g = @conllu.parse(gold) catch { _ => [] }
let p = @conllu.parse(pred) catch { _ => [] }
let result = @conllu.evaluate(g, p)
assert_eq(result.uas(), 50.0)
assert_eq(result.las(), 50.0)
}moon run cmd/main -- stats treebank.conllu # 句数/词数/多词/空节点/最长句/树库
moon run cmd/main -- validate treebank.conllu # 诊断列表,有 error 时退出码 1
moon run cmd/main -- eval gold.conllu pred.conllu
moon run cmd/main -- dump treebank.conllu # 归一化 dump,用于对拍
moon run cmd/main -- roundtrip treebank.conllu # 解析再序列化,校验无损D 2001
S 7 0 0 4
w 1 From ADP 3 casepython3 tools/fetch_corpus.py # 下载 UD 官方语料到 data/ud/(语料不随包分发)
python3 tools/diff_test.py --report| 文件 | 句数 | dump 行数 | 字段不一致 |
|---|---|---|---|
| en_ewt-ud-dev | 2001 | 27513 | 0 |
| en_ewt-ud-test | 2077 | 27528 | 0 |
| zh_gsd-ud-dev | 500 | 13166 | 0 |
| zh_gsd-ud-test | 500 | 12511 | 0 |
pub(all) suberror ConlluError {
ConlluError(Int, String)
}pub struct EvalResult {
sentence_pairs : Int
skipped_sentences : Int
gold_tokens : Int
aligned_tokens : Int
uas_correct : Int
las_correct : Int
}pub(all) enum RowKind {
Word
Multiword
Empty
}pub struct Stats {
sentences : Int
words : Int
multiwords : Int
empty_nodes : Int
comments : Int
max_length : Int
treebanks : Array[String]
}pub struct Token {
id_raw : String
form : String
lemma : String
upos : String
xpos : String
feats : String
head_raw : String
deprel : String
deps : String
misc : String
kind : RowKind
start : Int
end : Int
line : Int
}Install
Download zipCoNLL-U / Universal Dependencies toolchain: lossless parse and serialize, spec validation, tree queries, UAS/LAS evaluation, corpus statistics.