MoonBit Parquet reader/writer with row-group statistics and selective decoding; based on mizchi/parquet.
| Before, at the upstream baseline | This contribution |
|---|---|
| Every row group enters full reading | Single or multiple groups can be decoded directly |
| Statistics are not exposed | Raw Statistics and a validated signed-integer view are public |
| No public row-group inspection | Footer metadata, row groups and column chunks are inspectable |
| Consumers cannot select groups before decoding | Consumers can retain candidate groups and then filter actual rows |
let metadata = @parquet.read_metadata(bytes)
let group = metadata.row_groups()[2]
let statistics = group.columns()[0].signed_integer_statistics()
let batch = @parquet.read_row_groups_columnar(bytes, [2])git clone --branch feat/row-group-statistics https://github.com/sgy1023-crt/parquet.git
cd parquet
curl -fsSL https://cli.moonbitlang.com/install/unix.sh | bash -s -- "$(cat .moonbit-version)"
export MOON_HOME="$HOME/.moon"
export PATH="$MOON_HOME/bin:$PATH"
moon version --all
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -r scripts/row-group-requirements.txt
python scripts/prepare_dependency_cache.py --toolchain "$HOME/.moon"
python scripts/verify_delivery.py --toolchain "$HOME/.moon"moon run src/cmd/row_group_scan --target js -- fixtures/row-groups/base.parquet i32 102
moon run src/cmd/row_group_scan --target js -- fixtures/row-groups/base.parquet i64 9007199254740995moon add sgy1023-crt/parquet@0.3.0import {
"sgy1023-crt/parquet",
}just bench-compare-all
just bench-compare-rust| benchmark | moon(js) | moon(wasm-gc) | moon(native) | rust |
|---|---|---|---|---|
| read delta binary packed benchmark | 512.22 us | 127.01 us | 645.99 us | 203.85 us |
| read int32 null pages benchmark | 17.34 us | 10.85 us | 43.60 us | 7.61 us |
| read fixed length byte array benchmark | 87.17 us | 10.41 us | 54.68 us | unsupported |
| read alltypes plain benchmark | 20.49 us | 12.02 us | 45.42 us | 19.57 us |
| read alltypes dictionary benchmark | 16.83 us | 11.04 us | 42.53 us | 17.31 us |
| read int96 from spark benchmark | 10.90 us | 3.52 us | 16.19 us | 3.28 us |
| read empty snappy datapage v2 benchmark | 3.46 us | 1.72 us | 6.27 us | 2.58 us |
just # check + test
just target=js bench
just target=wasm-gc bench
just target=native bench
just bench-compare-all
just e2e-duckdb
moon infojust playground-install
just playground-devjust playground-build
just playground-build-pagespub struct Column {
name : String
column_type : ColumnType
repetition : Repetition
} derive(Eq, Debug)pub struct ColumnChunkMetadata {
path : ReadOnlyArray[String]
type_code : Int
logical_type : Bool
converted : Int?
flat : Bool
raw_statistics : Statistics?
values : Int64
compressed_size : Int64
uncompressed_size : Int64
page_offset : Int64
dictionary_offset : Int64?
order : ColumnOrder?
}fn ColumnChunkMetadata::signed_integer_statistics(self : ColumnChunkMetadata) -> SignedIntegerStatistics? raise ParquetErrorpub(all) enum ParquetColumnData {
Values(Array[Value])
BooleanValues(Array[Bool])
Int32Values(Array[Int])
NullableInt32Values(Array[Int?])
Int64Values(Array[Int64])
NullableInt64Values(Array[Int64?])
TimestampMicrosValues(Array[Int64])
NullableTimestampMicrosValues(Array[Int64?])
FloatValues(Array[Float])
DoubleValues(Array[Double])
Utf8Values(Array[Bytes])
NullableUtf8Values(Array[Bytes?])
BinaryValues(Array[Bytes])
NullableBinaryValues(Array[Bytes?])
} derive(Eq, Debug)impl Show for ParquetColumnDataimpl Eq for ParquetColumnarFileimpl Show for ParquetColumnarFilepub struct ParquetMetadata {
groups : ReadOnlyArray[RowGroupMetadata]
rows : Int64
writer : String?
}impl Show for Repetitionpub struct RowGroupMetadata {
group_index : Int
rows : Int64
chunks : ReadOnlyArray[ColumnChunkMetadata]
byte_size : Int64
compressed_size : Int64?
offset : Int64?
}pub struct SignedIntegerStatistics {
minimum : Int64?
maximum : Int64?
nulls : Int64?
}pub struct Statistics {
maximum : Bytes?
minimum : Bytes?
legacy_maximum : Bytes?
legacy_minimum : Bytes?
nulls : Int64?
distinct : Int64?
maximum_exact : Bool?
minimum_exact : Bool?
}fn new_parquet_columnar_file(columns : Array[Column], column_data : Array[ParquetColumnData], row_count : Int, created_by : String?) -> ParquetColumnarFilefn read_row_groups_columnar(data : Bytes, indices : Array[Int]) -> ParquetColumnarFile raise ParquetErrorInstall
Download zipMoonBit Parquet reader/writer with row-group statistics and selective decoding; based on mizchi/parquet.