parquet

    MoonBit Parquet reader/writer with row-group statistics and selective decoding; based on mizchi/parquet.

    moonbit
    parquet
    Download zip
    Version
    0.3.0
    License
    Apache-2.0
    Last updated
    6 hours ago
    Downloads
    5

    #MoonBit Parquet — row-group statistics and selective reading

    Parquet reader/writer for MoonBit, maintained here as sgy1023-crt/parquet with the original mizchi/parquet attribution and Apache-2.0 license preserved.

    #2026 MoonBit Hackathon — Row Group Statistics & Selective Decoding

    This is a substantive enhancement of mizchi/parquet, originally developed by mizchi, rather than a new Parquet implementation. sgy1023-crt contributed the row-group statistics, metadata and selective-decoding work in this fork, based on upstream commit 44d4dc9. The upstream consumer request, Issue #4, provides the use case. Existing reader/writer, codecs, DuckDB integration, playground and benchmarks remain upstream contributions. The browser bridge's scalar conversion preserves the behavior of the Apache-2.0 mizchi/duckdb 0.6.4 binding; the bridge now uses the existing npm DuckDB WASM client directly for compatibility with MoonBit 0.10.16.

    Before, at the upstream baselineThis contribution
    Every row group enters full readingSingle or multiple groups can be decoded directly
    Statistics are not exposedRaw Statistics and a validated signed-integer view are public
    No public row-group inspectionFooter metadata, row groups and column chunks are inspectable
    Consumers cannot select groups before decodingConsumers can retain candidate groups and then filter actual rows

    The three new APIs are read_metadata, read_row_group_columnar and read_row_groups_columnar. Existing read_bytes, read_bytes_columnar and file convenience APIs keep their signatures; full and selected reads share the existing decoders. See the public entries, metadata views, statistics implementation and API contract.

    let metadata = @parquet.read_metadata(bytes)
    let group = metadata.row_groups()[2]
    let statistics = group.columns()[0].signed_integer_statistics()
    let batch = @parquet.read_row_groups_columnar(bytes, [2])

    Selections preserve caller order. Duplicates and invalid indices fail before decoding; an empty selection retains the schema. The typed statistics view interprets only flat, unannotated INT32/INT64. Unknown semantics or ordering cannot justify skipping by value range; a validated all-NULL count can still eliminate a group. Raw fields preserve modern/legacy bounds, optional NULL/distinct counts and exactness flags separately.

    The separate consumer package and CLI implement column >= threshold: select candidate groups, decode them, then apply a real residual filter. For i32 >= 102, the base fixture selects group [2], decodes four rows and matches two. INT64 thresholds beyond 2^53 remain exact.

    Verified locally on 2026-10-19: JS, wasm, wasm-gc and native each passed 71/71 tests; 32 fixtures replayed byte-for-byte, comprising 16 valid files and 16 intentional negative cases. Four-backend instrumentation measured column-chunk decoder entries 8 → 2 for one of four groups and 0 for an empty selection. Poisoned-page tests independently show that unselected pages are not decoded. These are decoding-work measurements; speed, peak memory and file I/O benefits have not been measured. Input is still the whole file as Bytes; there is no Range I/O, predicate engine or column projection.

    Revalidated locally on 2026-10-30 with moonc 0.10.16: JS 74/74 (three additional browser conversion regressions), wasm / wasm-gc / native 71/71 each, all 17 delivery checks passed, and all 32 fixtures replayed. Decoder-entry counts remain 8/2/0/4/0. The production browser build also passed actual DuckDB WASM queries, including exact INT64, empty-result schema, NULL versus empty string, and recovery after an expected SQL error.

    #Reproduce this contribution

    Install the documented tools and native prerequisites: the MoonBit build fixed in .moonbit-version (moonc 0.10.16+b1a0e846e), Node.js 24 and CPython 3.13.5. The full native checks require a C compiler. DuckDB CLI 1.4.4 is used only by the separate interoperability check; no DuckDB native library is required. Use an isolated Python environment for fixture tools. A POSIX-shell example is:

    git clone --branch feat/row-group-statistics https://github.com/sgy1023-crt/parquet.git cd parquet curl -fsSL https://cli.moonbitlang.com/install/unix.sh | bash -s -- "$(cat .moonbit-version)" export MOON_HOME="$HOME/.moon" export PATH="$MOON_HOME/bin:$PATH" moon version --all python3 -m venv .venv . .venv/bin/activate python -m pip install -r scripts/row-group-requirements.txt python scripts/prepare_dependency_cache.py --toolchain "$HOME/.moon" python scripts/verify_delivery.py --toolchain "$HOME/.moon"

    The compiler and core must match this pinned build. It meets the acceptance guide's minimum compiler version of 0.10.16. The outdated MoonBit DuckDB / QuickCheck dependency chain has been removed; dependency source is not patched. Complete PowerShell commands are in DEPENDENCIES.md. The verification script runs tests, checks, builds, consumer smoke tests, fixture replay and real decoder-entry measurements, with a hard 60-second limit for each child check. It exits nonzero on failure. For a quick consumer run after setup:

    moon run src/cmd/row_group_scan --target js -- fixtures/row-groups/base.parquet i32 102 moon run src/cmd/row_group_scan --target js -- fixtures/row-groups/base.parquet i64 9007199254740995

    Read the validation record, delivery scope and hackathon submission notes. GitHub Actions run 41002594210 passed all nine active CI jobs for code commit 408c274e9560cad91eee1403b89b2094c295b536 with the pinned 0.10.16 toolchain, including checks, explicit builds, tests, macOS native and DuckDB interoperability. Fresh-checkout acceptance run 41002685678 also passed all 17 checks for that commit. Exact commands and raw receipts are attached to these runs; the final checklist records the remaining release step. The participant reports that this existing-project maintenance entry passed registration review. Final acceptance remains a separate assessment. See the maintenance schedule and final checklist for actual automation, evidence and the remaining manual procedures.

    The fork preserves the upstream Apache-2.0 license. Synthetic fixtures and their provenance are Apache-2.0; the upstream Apache test corpus retains its existing source and license information. Statistics follow a pinned Apache parquet-format specification.

    #Mooncakes package

    This fork uses sgy1023-crt/parquet@0.3.0 in this contributor's own namespace. Use MoonBit 0.10.16 or later; the exact tested build is pinned above. Add it to an existing MoonBit module:

    moon add sgy1023-crt/parquet@0.3.0

    Then import the library in the consuming package's moon.pkg:

    import {
    "sgy1023-crt/parquet",
    }

    Function and type contracts are preserved; the fork's import prefix differs from upstream mizchi/parquet. The GitHub final checklist records registry publication and independent consumer verification.

    #Status

    • Reader passes the vendored apache-parquet-testing fixtures used in this repo:
      • delta_binary_packed
      • delta_byte_array
      • delta_encoding_optional_column
      • delta_encoding_required_column
      • int32_with_null_pages
      • fixed_length_byte_array
      • datapage_v2_empty_datapage.snappy.parquet
      • int96_from_spark
      • alltypes_plain
      • alltypes_dictionary
    • Writer currently supports flat schemas with Int32, Int64, String, and Binary, with Required / Optional repetition.
    • DuckDB interoperability is checked both ways:
      • DuckDB-written parquet can be read by this implementation.
      • MoonBit-written parquet can be read by DuckDB.

    #Benchmark

    The following upstream benchmarks predate this contribution and do not measure the new selective-reading APIs.

    Measured on 2026-03-09 on Apple M5, macOS 26.2, Darwin arm64.

    Reproduce:

    just bench-compare-all just bench-compare-rust

    Comparison table from just bench-compare-all:

    benchmarkmoon(js)moon(wasm-gc)moon(native)rust
    read delta binary packed benchmark512.22 us127.01 us645.99 us203.85 us
    read int32 null pages benchmark17.34 us10.85 us43.60 us7.61 us
    read fixed length byte array benchmark87.17 us10.41 us54.68 usunsupported
    read alltypes plain benchmark20.49 us12.02 us45.42 us19.57 us
    read alltypes dictionary benchmark16.83 us11.04 us42.53 us17.31 us
    read int96 from spark benchmark10.90 us3.52 us16.19 us3.28 us
    read empty snappy datapage v2 benchmark3.46 us1.72 us6.27 us2.58 us

    Notes:

    • Rust is the local baseline in tools/rust-bench, built on top of Apache's parquet crate.
    • fixed_length_byte_array is marked unsupported because the current Rust benchmark tool does not read that fixture.
    • On this machine, wasm-gc is the fastest MoonBit target for the read-side microbenchmarks above.

    #Development

    just # check + test just target=js bench just target=wasm-gc bench just target=native bench just bench-compare-all just e2e-duckdb moon info

    #Browser Playground

    The repository includes a browser playground that generates parquet on DuckDB WASM and lets you try SQL against it immediately.

    just playground-install just playground-dev

    Production build:

    just playground-build just playground-build-pages

    In the UI, edit schema JSON / rows JSON / SQL, generate parquet bytes with this Parquet implementation, and execute read_parquet('playground.parquet') on the existing npm DuckDB WASM client through the MoonBit playground bridge.

    GitHub Pages:

    https://mizchi.github.io/parquet/

    #License

    Apache-2.0

    ParquetError

    pub(all) suberror ParquetError {
    InvalidData(String)
    Unsupported(String)
    Io(String)
    } derive(Eq,
    Debug
    )

    ParquetError::equal

    ParquetError::not_equal

    fn ParquetError::not_equal(x : ParquetError, y : ParquetError) -> Bool

    ParquetError::output

    fn ParquetError::output(self : ParquetError, logger : &Logger) -> Unit

    ParquetError::to_string

    fn ParquetError::to_string(self : ParquetError) -> String

    Column

    pub struct Column {
    name : String
    column_type : ColumnType
    repetition : Repetition
    } derive(Eq,
    Debug
    )

    impl Show for Column

    Column::column_type

    fn Column::column_type(self : Column) -> ColumnType

    Column::equal

    fn Column::equal(Column, Column) -> Bool

    Column::name

    fn Column::name(self : Column) -> String

    Column::not_equal

    fn Column::not_equal(x : Column, y : Column) -> Bool

    Column::output

    fn Column::output(self : Column, logger : &Logger) -> Unit

    Column::repetition

    fn Column::repetition(self : Column) -> Repetition

    Column::to_repr

    Column::to_string

    fn Column::to_string(self : Column) -> String

    ColumnChunkMetadata

    pub struct ColumnChunkMetadata {
    path : ReadOnlyArray[String]
    type_code : Int
    logical_type : Bool
    converted : Int?
    flat : Bool
    raw_statistics : Statistics?
    values : Int64
    compressed_size : Int64
    uncompressed_size : Int64
    page_offset : Int64
    dictionary_offset : Int64?
    order : ColumnOrder?
    }

    ColumnChunkMetadata::column_order

    ColumnChunkMetadata::converted_type

    fn ColumnChunkMetadata::converted_type(self : ColumnChunkMetadata) -> Int?

    ColumnChunkMetadata::data_page_offset

    fn ColumnChunkMetadata::data_page_offset(self : ColumnChunkMetadata) -> Int64

    ColumnChunkMetadata::dictionary_page_offset

    fn ColumnChunkMetadata::dictionary_page_offset(self : ColumnChunkMetadata) -> Int64?

    ColumnChunkMetadata::has_logical_type

    fn ColumnChunkMetadata::has_logical_type(self : ColumnChunkMetadata) -> Bool

    ColumnChunkMetadata::is_flat

    fn ColumnChunkMetadata::is_flat(self : ColumnChunkMetadata) -> Bool

    ColumnChunkMetadata::num_values

    fn ColumnChunkMetadata::num_values(self : ColumnChunkMetadata) -> Int64

    ColumnChunkMetadata::path_in_schema

    fn ColumnChunkMetadata::path_in_schema(self : ColumnChunkMetadata) -> ReadOnlyArray[String]

    ColumnChunkMetadata::physical_type

    ColumnChunkMetadata::signed_integer_statistics

    fn ColumnChunkMetadata::signed_integer_statistics(self : ColumnChunkMetadata) -> SignedIntegerStatistics? raise ParquetError

    Only flat, unannotated INT32/INT64 have validated signed semantics here. Missing/unknown column order disables bounds, but preserves a known null count. Legacy bounds are never substituted. exact=false still permits safe bounds.

    ColumnChunkMetadata::statistics

    ColumnChunkMetadata::total_compressed_size

    fn ColumnChunkMetadata::total_compressed_size(self : ColumnChunkMetadata) -> Int64

    ColumnChunkMetadata::total_uncompressed_size

    fn ColumnChunkMetadata::total_uncompressed_size(self : ColumnChunkMetadata) -> Int64

    ColumnOrder

    pub(all) enum ColumnOrder {
    TypeDefined
    Unknown(Int)
    } derive(Eq,
    Debug
    )

    ColumnOrder::equal

    fn ColumnOrder::equal(ColumnOrder, ColumnOrder) -> Bool

    ColumnOrder::not_equal

    fn ColumnOrder::not_equal(x : ColumnOrder, y : ColumnOrder) -> Bool

    ColumnType

    pub(all) enum ColumnType {
    Boolean
    Int32
    Int64
    TimestampMicros
    Float
    Double
    String
    Binary
    } derive(Eq,
    Debug
    )

    impl Show for ColumnType

    ColumnType::equal

    fn ColumnType::equal(ColumnType, ColumnType) -> Bool

    ColumnType::not_equal

    fn ColumnType::not_equal(x : ColumnType, y : ColumnType) -> Bool

    ColumnType::output

    fn ColumnType::output(self : ColumnType, logger : &Logger) -> Unit

    ColumnType::to_string

    fn ColumnType::to_string(self : ColumnType) -> String

    ParquetColumnData

    pub(all) enum ParquetColumnData {
    Values(Array[Value])
    BooleanValues(Array[Bool])
    Int32Values(Array[Int])
    NullableInt32Values(Array[Int?])
    Int64Values(Array[Int64])
    NullableInt64Values(Array[Int64?])
    TimestampMicrosValues(Array[Int64])
    NullableTimestampMicrosValues(Array[Int64?])
    FloatValues(Array[Float])
    DoubleValues(Array[Double])
    Utf8Values(Array[Bytes])
    NullableUtf8Values(Array[Bytes?])
    BinaryValues(Array[Bytes])
    NullableBinaryValues(Array[Bytes?])
    } derive(Eq,
    Debug
    )

    ParquetColumnData::equal

    ParquetColumnData::length

    fn ParquetColumnData::length(self : ParquetColumnData) -> Int

    ParquetColumnData::not_equal

    fn ParquetColumnData::not_equal(x : ParquetColumnData, y : ParquetColumnData) -> Bool

    ParquetColumnData::output

    fn ParquetColumnData::output(self : ParquetColumnData, logger : &Logger) -> Unit

    ParquetColumnData::to_string

    fn ParquetColumnData::to_string(self : ParquetColumnData) -> String

    ParquetColumnarFile

    pub struct ParquetColumnarFile {
    columns : Array[Column]
    column_data : Array[ParquetColumnData]
    row_count : Int
    column_values_cache : Array[Array[Value]]?
    created_by : String?
    } derive(
    Debug
    )

    ParquetColumnarFile::column_data

    ParquetColumnarFile::column_values

    ParquetColumnarFile::columns

    ParquetColumnarFile::created_by

    fn ParquetColumnarFile::created_by(self : ParquetColumnarFile) -> String?

    ParquetColumnarFile::equal

    ParquetColumnarFile::not_equal

    ParquetColumnarFile::output

    fn ParquetColumnarFile::output(self : ParquetColumnarFile, logger : &Logger) -> Unit

    ParquetColumnarFile::row_count

    fn ParquetColumnarFile::row_count(self : ParquetColumnarFile) -> Int

    ParquetColumnarFile::to_string

    fn ParquetColumnarFile::to_string(self : ParquetColumnarFile) -> String

    ParquetFile

    pub struct ParquetFile {
    columns : Array[Column]
    row_count : Int
    column_data : Array[ParquetColumnData]
    rows_cache : Array[Array[Value]]?
    created_by : String?
    } derive(
    Debug
    )

    impl Eq for ParquetFile
    impl Show for ParquetFile

    ParquetFile::column_data

    fn ParquetFile::column_data(self : ParquetFile) -> Array[ParquetColumnData]

    ParquetFile::columns

    fn ParquetFile::columns(self : ParquetFile) -> Array[Column]

    ParquetFile::created_by

    fn ParquetFile::created_by(self : ParquetFile) -> String?

    ParquetFile::equal

    fn ParquetFile::equal(self : ParquetFile, other : ParquetFile) -> Bool

    ParquetFile::not_equal

    fn ParquetFile::not_equal(x : ParquetFile, y : ParquetFile) -> Bool

    ParquetFile::output

    fn ParquetFile::output(self : ParquetFile, logger : &Logger) -> Unit

    ParquetFile::row_count

    fn ParquetFile::row_count(self : ParquetFile) -> Int

    ParquetFile::rows

    fn ParquetFile::rows(self : ParquetFile) -> Array[Array[Value]]

    ParquetFile::to_string

    fn ParquetFile::to_string(self : ParquetFile) -> String

    ParquetMetadata

    pub struct ParquetMetadata {
    groups : ReadOnlyArray[RowGroupMetadata]
    rows : Int64
    writer : String?
    }

    ParquetMetadata::created_by

    fn ParquetMetadata::created_by(self : ParquetMetadata) -> String?

    ParquetMetadata::row_count

    fn ParquetMetadata::row_count(self : ParquetMetadata) -> Int64

    ParquetMetadata::row_group_count

    fn ParquetMetadata::row_group_count(self : ParquetMetadata) -> Int

    ParquetMetadata::row_groups

    fn ParquetMetadata::row_groups(self : ParquetMetadata) -> ReadOnlyArray[RowGroupMetadata]

    PhysicalType

    pub(all) enum PhysicalType {
    Boolean
    Int32
    Int64
    Int96
    Float
    Double
    ByteArray
    FixedLenByteArray
    Unknown(Int)
    } derive(Eq,
    Debug
    )

    PhysicalType::equal

    PhysicalType::not_equal

    fn PhysicalType::not_equal(x : PhysicalType, y : PhysicalType) -> Bool

    Repetition

    pub(all) enum Repetition {
    Required
    Optional
    Repeated
    } derive(Eq,
    Debug
    )

    impl Show for Repetition

    Repetition::equal

    fn Repetition::equal(Repetition, Repetition) -> Bool

    Repetition::not_equal

    fn Repetition::not_equal(x : Repetition, y : Repetition) -> Bool

    Repetition::output

    fn Repetition::output(self : Repetition, logger : &Logger) -> Unit

    Repetition::to_string

    fn Repetition::to_string(self : Repetition) -> String

    RowGroupMetadata

    pub struct RowGroupMetadata {
    group_index : Int
    rows : Int64
    chunks : ReadOnlyArray[ColumnChunkMetadata]
    byte_size : Int64
    compressed_size : Int64?
    offset : Int64?
    }

    RowGroupMetadata::columns

    fn RowGroupMetadata::columns(self : RowGroupMetadata) -> ReadOnlyArray[ColumnChunkMetadata]

    RowGroupMetadata::file_offset

    fn RowGroupMetadata::file_offset(self : RowGroupMetadata) -> Int64?

    RowGroupMetadata::index

    fn RowGroupMetadata::index(self : RowGroupMetadata) -> Int

    RowGroupMetadata::row_count

    fn RowGroupMetadata::row_count(self : RowGroupMetadata) -> Int64

    RowGroupMetadata::total_byte_size

    fn RowGroupMetadata::total_byte_size(self : RowGroupMetadata) -> Int64

    RowGroupMetadata::total_compressed_size

    fn RowGroupMetadata::total_compressed_size(self : RowGroupMetadata) -> Int64?

    SignedIntegerStatistics

    pub struct SignedIntegerStatistics {
    minimum : Int64?
    maximum : Int64?
    nulls : Int64?
    }

    Signed physical integer bounds. They may be conservative bounds rather than observed extrema, even when the writer omits the exactness flags.

    SignedIntegerStatistics::max_value

    fn SignedIntegerStatistics::max_value(self : SignedIntegerStatistics) -> Int64?

    SignedIntegerStatistics::min_value

    fn SignedIntegerStatistics::min_value(self : SignedIntegerStatistics) -> Int64?

    SignedIntegerStatistics::null_count

    fn SignedIntegerStatistics::null_count(self : SignedIntegerStatistics) -> Int64?

    Statistics

    pub struct Statistics {
    maximum : Bytes?
    minimum : Bytes?
    legacy_maximum : Bytes?
    legacy_minimum : Bytes?
    nulls : Int64?
    distinct : Int64?
    maximum_exact : Bool?
    minimum_exact : Bool?
    }

    Raw footer statistics. Modern bounds and deprecated legacy bounds are kept separately. A missing count or exactness flag remains None.

    Statistics::distinct_count

    fn Statistics::distinct_count(self : Statistics) -> Int64?

    Statistics::is_max_value_exact

    fn Statistics::is_max_value_exact(self : Statistics) -> Bool?

    Statistics::is_min_value_exact

    fn Statistics::is_min_value_exact(self : Statistics) -> Bool?

    Statistics::legacy_max

    fn Statistics::legacy_max(self : Statistics) -> Bytes?

    Statistics::legacy_min

    fn Statistics::legacy_min(self : Statistics) -> Bytes?

    Statistics::max_value

    fn Statistics::max_value(self : Statistics) -> Bytes?

    Statistics::min_value

    fn Statistics::min_value(self : Statistics) -> Bytes?

    Statistics::null_count

    fn Statistics::null_count(self : Statistics) -> Int64?

    Value

    pub(all) enum Value {
    Boolean(Bool)
    Int32(Int)
    Int64(Int64)
    TimestampMicros(Int64)
    Float(Float)
    Double(Double)
    String(String)
    Binary(Bytes)
    Null
    } derive(Eq,
    Debug
    )

    impl Show for Value

    Value::equal

    fn Value::equal(Value, Value) -> Bool

    Value::not_equal

    fn Value::not_equal(x : Value, y : Value) -> Bool

    Value::output

    fn Value::output(self : Value, logger : &Logger) -> Unit

    Value::to_repr

    Value::to_string

    fn Value::to_string(self : Value) -> String

    new_column

    fn new_column(name : String, column_type : ColumnType, repetition : Repetition) -> Column

    new_parquet_columnar_file

    fn new_parquet_columnar_file(columns : Array[Column], column_data : Array[ParquetColumnData], row_count : Int, created_by : String?) -> ParquetColumnarFile

    new_parquet_file

    fn new_parquet_file(columns : Array[Column], rows : Array[Array[Value]], created_by : String?) -> ParquetFile

    read_bytes

    fn read_bytes(data : Bytes) -> ParquetFile raise ParquetError

    Read a parquet document from bytes.

    read_bytes_columnar

    fn read_bytes_columnar(data : Bytes) -> ParquetColumnarFile raise ParquetError

    Read a parquet document from bytes without row materialization.

    read_file

    fn read_file(path : String) -> ParquetFile raise ParquetError

    Read a parquet file from disk.

    read_file_columnar

    fn read_file_columnar(path : String) -> ParquetColumnarFile raise ParquetError

    Read a parquet file from disk without row materialization.

    read_metadata

    fn read_metadata(data : Bytes) -> ParquetMetadata raise ParquetError

    Inspect footer metadata without decoding column chunks. Byte input is still the whole file; this API does not perform range I/O.

    read_row_group_columnar

    fn read_row_group_columnar(data : Bytes, index : Int) -> ParquetColumnarFile raise ParquetError

    Decode one row group by its zero-based file index.

    read_row_groups_columnar

    fn read_row_groups_columnar(data : Bytes, indices : Array[Int]) -> ParquetColumnarFile raise ParquetError

    Decode only the requested row groups, in caller order. Duplicate or invalid indices are errors, checked before decoding. An empty selection keeps schema.

    write_bytes

    fn write_bytes(file : ParquetFile) -> Bytes raise ParquetError

    Encode a parquet document to bytes.

    The current writer supports flat schemas of Int32, Int64, String, and Binary, with Required or Optional repetition.

    write_file

    fn write_file(path : String, file : ParquetFile) -> Unit raise ParquetError

    Encode a parquet document and write it to disk.

    Powered by MoonBit

    Site sourceReport issueModulesSkillsBuild queueStatistics

    © 2026 mooncakes.io