MoonBit-native Delta Lake table-format kernel: transaction log replay, snapshots, time travel, scan planning and append writes.
生态里已经有 Parquet 编解码、DataFrame、对 Parquet 跑 SQL 的查询引擎、Iceberg 只读——缺的是"带事务日志的表"这一层。本项目补这一层。参赛申报书:申报书.md | 第三方声明:THIRD_PARTY_NOTICES.md已发布:mooncakes.io/docs/xiguaAp6y3/moondelta-core(0.3.1;空工程安装日志见 artifacts/acceptance/mooncakes-0.3.1/)。registry 版本与当前源码一致。
# 从 mooncakes.io 安装当前发布版本 0.3.1
moon add xiguaAp6y3/moondelta-core
# 或从源码构建
moon check --target native
moon test --target native
# CLI(源码根目录为 src/,因此包路径带 src/ 前缀)
moon run --target native src/cmd/delta -- inspect fixtures/delta/table_a
moon run --target native src/cmd/delta -- history fixtures/delta/table_a
moon run --target native src/cmd/delta -- scan fixtures/delta/table_a --where "id>=60"
moon run --target native src/cmd/delta -- read fixtures/delta/covid --limit 5table : fixtures/delta/table_a
current version : 12 (log versions: 13)
files : 120 rows: 120 bytes: 54232
protocol : reader=1 writer=1
format : parquet
partition columns: (none)
schema:
id: int
last checkpoint : version=10
note : checkpoint present; truncated / checkpoint-only tables are recovered from the latest single-file checkpoint注:样例中的 reader=1 writer=1 是 delta-rs fixture 自身的协议字段;本项目写出的表为 minReaderVersion = 1, minWriterVersion = 2。
let table = @delta.DeltaTable::open("path/to/table")
let snapshot = table.snapshot() // 最新版本
let old = table.snapshot_at(1) // 时间旅行
let report = @delta.scan(snapshot, {
partitions: [("dt", "2026-10-18")],
predicate: Some({ column: "id", op: GeV, value: 60.0 }),
})
let rows = @delta.read_snapshot("path/to/table", snapshot, Some(100))
for c in rows.columns {
println(c.name + " : " + @delta.delta_type_name(c.type_))
}@delta.create_table(root, schema_json, [])
let columns : Array[@delta.ReadColumn] = [
{ name: "id", type_: Int64Type, cells: [CellI64(1L), CellI64(2L)] },
{ name: "name", type_: StringType, cells: [CellStr("a"), CellStr("b")] },
]
@delta.append_rows(root, columns, 1750000000000L) // 写 parquet + 提交新版本| 维度 | 支持 | 不支持 / 显式拒绝 |
|---|---|---|
| protocol | reader v1 读取;writer v2 写出 | reader > 1、readerFeatures、非空 writerFeatures;未声称任意 reader v1 表全部兼容 |
| 日志 | version 0 起连续 JSON commit;20 位文件名;截断 / checkpoint-only 从单文件 checkpoint 恢复 | 多部分 checkpoint;checkpoint 缺失 protocol/metaData;checkpoint 中真实出现的未知 action |
| schema | 11 种 primitive 与 struct / 3-level list / map 读写;flat 与 nested 叶均校验 Delta 类型和 Parquet 物理类型 | decimal、uuid、timestamp_ntz、timestamp millis/nanos |
| 压缩 | UNCOMPRESSED、SNAPPY | ZSTD、GZIP、BROTLI、LZ4 |
| 写入类型 | boolean / byte / short / int / long / float / double / string / binary / date / timestamp,包含嵌套叶 | decimal、uuid、timestamp_ntz、timestamp millis/nanos |
| 写入表 | 非分区 append;校验列名/类型/NOT NULL;put-if-absent | 分区 append;列在 Parquet 中恒为 Optional |
| 删除向量 | JSON 与 checkpoint 的 add.deletionVector 均拒绝 | column mapping、v2 checkpoint、CDF、MERGE/UPDATE/DELETE |
| 存储 | 本地文件系统 | 对象存储(S3/R2) |
moon check --target native
moon test --target native
moon fmtpub(all) enum Action {
ProtocolAction(Protocol)
MetaDataAction(TableInfo)
AddAction(AddFile)
RemoveAction(RemoveFile)
CommitInfoAction(CommitInfo)
TxnAction(Txn)
IgnoredAction(String)
} derive(Eq, Debug)pub(all) struct DeltaTable {
root : String
}pub(all) enum DeltaType {
Int32Type
Int64Type
FloatType
DoubleType
BooleanType
StringType
BinaryType
ByteType
ShortType
DateType
TimestampType
DecimalType(precision~ : Int, scale~ : Int)
NestedType(kind~ : String, fields~ : Array[DeltaField])
UnknownType(raw~ : String)
} derive(Eq, Debug)pub(all) struct ReadResult {
columns : Array[ReadColumn]
row_count : Int
files_read : Int
} derive(Debug)pub(all) struct ScanFilter {
partitions : Array[(String, String)]
predicate : ColumnPredicate?
} derive(Eq, Debug)pub(all) struct TableInfo {
id : String
name : String?
schema : DeltaSchema
partition_columns : Array[String]
format_provider : String
configuration : Array[(String, String)]
} derive(Eq, Debug)fn append_rows(root : String, columns : Array[ReadColumn], timestamp : Int64) -> Int raise DeltaErrorfn create_table(root : String, schema_json : String, partition_columns : Array[String]) -> Int raise DeltaErrorfn has_checkpoint_file(root : String) -> Boolfn read_snapshot_shredded(root : String, snapshot : Snapshot, limit : Int?) -> ReadResult raise DeltaErrorfn shred_rows(schema : DeltaSchema, leaves : Array[LeafColumn], row_count : Int) -> Array[Array[Cell]] raise DeltaErrorfn write_nested_parquet_bytes(schema : DeltaSchema, columns : Array[ReadColumn], row_count : Int) -> Bytes raise DeltaErrorInstall
Download zipMoonBit-native Delta Lake table-format kernel: transaction log replay, snapshots, time travel, scan planning and append writes.