🥮 mooncakes.io
    Sign in

    arrow

    MoonArrow: pure MoonBit Arrow IPC stream and file reader/writer with checked metadata and PyArrow interoperability.

    arrow
    ipc
    columnar
    serialization
    Download zip
    Author
    Version
    0.2.1
    License
    MIT
    Repository
    Last updated
    25 days ago
    Downloads
    63

    No dependencies

    #MoonArrow · 月矢

    shunge/arrow 是纯 MoonBit 实现的 Apache Arrow IPC 读写库，让 MoonBit 程序通过标准列式数据格式与 Python 等数据工具交换数据。 本文对应 0.2.1，模块名为 shunge/arrow。0.1.0 只有初始七种基础类型， 以下扩展能力请使用 shunge/arrow@0.2.1。

    生产代码只依赖 MoonBit 标准库，不依赖 PyArrow、Arrow C++ 或 FFI。 PyArrow 仅用于开发时的独立互操作测试。

    #已实现的范围

    能力当前状态
    列类型Null、Boolean、各宽度有/无符号整数、Float32/64、Date32/64、Timestamp、Duration、Utf8/Binary、LargeUtf8/LargeBinary、FixedSizeBinary
    复合类型List、LargeList、FixedSizeList、Struct、Map、顶层 Int32 索引的 Utf8/Binary Dictionary
    空值有效位图、null count、非空字段检查
    IPC streamV5 写入；V4/V5 读取；兼容 legacy 前缀
    IPC fileARROW1 文件、footer 索引、多批次与随机批次读取
    元数据schema 与 field 的 UTF-8 key/value，保留顺序、重复键与缺省值
    读取方式整体读入、在已有 Bytes 上逐批解码，或增量推入 stream 字节块
    数据处理BatchBuilder、RecordBatch/Table 的投影、筛选、拼接、重分块、摘要
    后端Native、JS、Wasm、Wasm-GC 均有测试

    #快速示例

    以下代码块会由 MoonBit 文档测试执行。在本仓库中运行不需要先发布包。 在其他 MoonBit 项目中安装 0.2.1：

    moon add shunge/arrow@0.2.1

    然后在调用方的 moon.pkg 中导入 shunge/arrow：

    import {
  "shunge/arrow",
}

    Mooncakes 模块页面 · GitHub 源码

    本仓库的 examples/consumer 是一个独立模块，moon.work 将它的 shunge/arrow@0.2.1 依赖解析到本地源码，因此可验证本版公开 API。 运行 moon run examples/consumer --target native。

    ///|
    test "nullable Arrow file roundtrip" {
      let schema = @arrow.Schema::new([
        @arrow.Field::new("id", Int64, nullable=false),
        @arrow.Field::new("name", Utf8),
      ])
      let batch = @arrow.RecordBatch::new(schema, [
        Int64s([Some(1L), Some(9007199254740993L)]),
        Strings([Some("月兔"), None]),
      ])
      let bytes = @arrow.write_file(schema, [batch])
      let reader = @arrow.FileReader::new(bytes)
      assert_eq(reader.num_batches(), 1)
      let restored = reader.get_batch(0)
      assert_true(
        restored.columns[0] is Int64s([Some(1L), Some(9007199254740993L)]),
      )
      assert_true(restored.columns[1] is Strings([Some("月兔"), None]))
    }

    命令行演示：

    moon run cmd/main --target native

    write_stream / read_stream 对应 IPC stream；write_file / read_file 对应 IPC file。StreamReader::next() 每次解码一个批次，结束时返回 None， 解析报错后也进入结束状态。FileReader::get_batch(index) 按 footer 索引读取指定批次。 StreamReader 和 FileReader 保留完整输入 Bytes。 IncrementalReader::push 接受任意切分的 stream 字节块，复制尚未处理完的帧， 返回拥有数据的已完成批次；schema() 返回防御性副本。输入结束后调用 finish() 检查截断；完整消息边界处即使没有 EOS 标记也可正常结束。 解析或结束检查出错后需创建新 reader，正常结束后不能再推入非空数据。 IncrementalWriter 与 FileWriter 都按 start()、逐批 write_batch()、 finish() 的顺序返回 Bytes，调用方按顺序写入目标。它们复制传入的 schema； stream writer 保留字典状态，file writer 还保留 footer 索引。批次写入或收尾失败后 需创建新 writer。这些 API 都不执行文件或网络 I/O，传输与落盘由调用方负责。

    BatchBuilder::append_row 接受按 schema 顺序排列的 Value?； Table::filter_where 使用类型化 Predicate，空值在 Eq/Ne 中均不匹配。 Table::to_file 和 Table::from_file 可衔接 IPC。典型外部消费代码见 独立消费模块；两个 PyArrow 文件工作流见 tools/workflow.py。

    #输入检查与限制

    API 抛出 ArrowError::Invalid、Unsupported 或 LimitExceeded。 读入时检查长度、偏移、位图与空值数量、UTF-8、schema 和批次布局。 写入前会递归重新检查数组长度、类型、schema 一致性、非空约束、数值范围、 Date64 整日值及固定二进制宽度；footer 版本不能早于文件头版本。 公开数组可变；调用方应避免在 reader 使用期间修改其 schema。

    可通过 ReadLimits 设置每批行数、每批值数、字段数、批次数、元数据字节数、 body 字节数、递归深度、字典值数和字典消息数上限。默认每批最多 100 万行、 800 万个值、1024 字段，最多 10000 批次，元数据 16 MiB、body 256 MiB、 递归深度 64、字典值 800 万、字典消息 10000。 这些是格式解析预算，并非精确的进程内存上限；整体读入多个批次会累积内存。

    #兼容性验证

    moon check --target all --deny-warn
moon test --target all
python -m pip install -r tools/requirements-interop.txt
python tools/interop.py --target native
python tools/interop.py --target js
python tools/workflow.py --target native
python tools/count_core.py --min-effective 4001

    当前工具链的 moon test 已包含文档测试，不需要已弃用的 --doc 参数。 互操作测试会让 PyArrow 生成输入、让 MoonBit 解码并重写，再由 PyArrow 读取和完整校验。另有一个 PyArrow 生成的固定样本，直接由四个后端运行。 测试覆盖空批次、位图字节边界、Unicode/BOM/NUL、切片、数值极值、 时间单位、嵌套布局、字典和浮点位模式。测试驱动用十六进制命令行参数 传输小样本，不是大文件工具。

    #尚未支持

    Decimal、Union、RunEndEncoded、View 类型、压缩 body、big-endian schema、 非 Int32 索引或嵌套的 Dictionary、message/footer 自定义元数据尚未支持。 schema/field 的 extension metadata 会原样保留，但不解释扩展类型语义。

    目前解码生成拥有数据的数组，不提供零拷贝视图、Arrow C Data Interface 或并行计算。仅支持类型范围内的未压缩 Feather V2；不能读取所有 .feather 文件。文件布局方面，当前 reader 要求批次连续排列。

    详细约束见 格式契约，后续工作见 路线图。

    #格式依据与许可

    实现依据 Apache Arrow Columnar Format 及其 format schemas。 元数据编码器仅覆盖本项目使用的 Arrow 表结构，不是通用 FlatBuffers 库。 代码采用 MIT 许可；本项目为独立实现，不代表 Apache 软件基金会。

    ArrowError

    source
    pub(all) suberror ArrowError {
      Invalid(String)
      Unsupported(String)
      LimitExceeded(String)
    } derive(
    Debug    )

    Errors for malformed input, unsupported features, and invalid batches.

    ArrowError::to_repr

    source

    BatchBuilder

    source
    pub struct BatchBuilder {
      // private fields
    }

    Row-wise builder for an immutable schema snapshot.

    BatchBuilder::append_batch

    source
    fn BatchBuilder::append_batch(self : BatchBuilder, batch : RecordBatch) -> Unit raise ArrowError

    Append a validated batch to a row builder. Dictionary fields are currently unsupported by this builder and are rejected before any row is appended.

    BatchBuilder::append_row

    source
    fn BatchBuilder::append_row(self : BatchBuilder, values : Array[Value?]) -> Unit raise ArrowError

    Append one complete row. A rejected row leaves this builder unchanged.

    BatchBuilder::finish

    source
    fn BatchBuilder::finish(self : BatchBuilder) -> RecordBatch raise ArrowError

    Return a snapshot. Later appends do not change previously finished batches.

    BatchBuilder::new

    source
    fn BatchBuilder::new(schema : Schema) -> BatchBuilder raise ArrowError

    BatchBuilder::num_rows

    source
    fn BatchBuilder::num_rows(self : BatchBuilder) -> Int

    Column

    source
    pub(all) enum Column {
      Nulls(Int)
      Booleans(Array[Bool?])
      Int8s(Array[Int?])
      Int16s(Array[Int?])
      Int32s(Array[Int?])
      Int64s(Array[Int64?])
      UInt8s(Array[Int?])
      UInt16s(Array[Int?])
      UInt32s(Array[UInt?])
      UInt64s(Array[UInt64?])
      Float32s(Array[Float?])
      Float64s(Array[Double?])
      Date32s(Array[Int?])
      Date64s(Array[Int64?])
      Timestamps(Array[Int64?], TimeUnit, String?)
      Durations(Array[Int64?], TimeUnit)
      Lists(Array[Int], Array[Bool], Column)
      LargeLists(Array[Int], Array[Bool], Column)
      FixedLists(Array[Bool], Column, Int)
      Maps(Array[Int], Array[Bool], Column, Bool)
      Structs(Array[Bool], Array[Column])
      Dictionaries(Array[Int?], Column, Int64, Bool)
      Strings(Array[String?])
      Binaries(Array[Bytes?])
      LargeStrings(Array[String?])
      LargeBinaries(Array[Bytes?])
      FixedBinaries(Array[Bytes?], Int)
    } derive(
    Debug    )

    Owned, decoded column values. None represents an Arrow null slot. Binary data is never interpreted as text; Int64 values retain all 64 bits.

    Column::data_type

    source
    fn Column::data_type(self : Column) -> DataType

    Column::get

    source
    fn Column::get(self : Column, index : Int) -> Value? raise ArrowError

    Return a logical scalar, resolving dictionary indices and nested child arrays. A returned None represents a null slot at this level.

    Column::length

    source
    fn Column::length(self : Column) -> Int

    Column::null_count

    source
    fn Column::null_count(self : Column) -> Int

    Column::slice

    source
    fn Column::slice(self : Column, start : Int, length : Int) -> Column raise ArrowError

    Column::summary

    source
    fn Column::summary(self : Column) -> ColumnSummary

    Column::take

    source
    fn Column::take(self : Column, indices : Array[Int]) -> Column raise ArrowError

    Copy selected column slots in arbitrary order. Duplicate indices are kept.

    Column::to_repr

    source

    ColumnRange

    source
    pub(all) enum ColumnRange {
      NoRange
      IntRange(Int, Int)
      Int64Range(Int64, Int64)
      UIntRange(UInt, UInt)
      UInt64Range(UInt64, UInt64)
      Float32Range(Float, Float)
      Float64Range(Double, Double)
      ByteLengthRange(Int, Int, Int64)
      BooleanCounts(Int, Int)
    } derive(
    Debug    )

    Physical value range for a column. Float ranges ignore NaN values; all null slots are excluded. Byte lengths count encoded UTF-8 bytes for text.

    ColumnRange::to_repr

    source

    ColumnSummary

    source
    pub(all) struct ColumnSummary {
      length : Int
      null_count : Int
      range : ColumnRange
    } derive(
    Debug    )

    ColumnSummary::combine

    source
    fn ColumnSummary::combine(self : ColumnSummary, other : ColumnSummary) -> ColumnSummary raise ArrowError

    Combine summaries of batches with the same column type. NoRange acts as an identity for an all-null batch, while a type mismatch is an explicit error.

    ColumnSummary::to_repr

    source

    DataType

    source
    pub(all) enum DataType {
      Null
      Boolean
      Int8
      Int16
      Int32
      Int64
      UInt8
      UInt16
      UInt32
      UInt64
      Float32
      Float64
      Date32
      Date64
      Timestamp(TimeUnit, String?)
      Duration(TimeUnit)
      List
      LargeList
      FixedSizeList(Int)
      Map(Bool)
      Struct
      Dictionary(Int64, DataType, Bool)
      Utf8
      Binary
      LargeUtf8
      LargeBinary
      FixedSizeBinary(Int)
    } derive(Eq, 
    Debug    )

    Data types supported by the current Arrow IPC subset.

    DataType::equal

    source
    fn DataType::equal(DataType, DataType) -> Bool

    DataType::not_equal

    source
    fn DataType::not_equal(x : DataType, y : DataType) -> Bool

    DataType::to_repr

    source

    Field

    source
    pub(all) struct Field {
      name : String
      data_type : DataType
      nullable : Bool
      metadata : Array[KeyValue]
      children : Array[Field]
    } derive(Eq, 
    Debug    )

    A named field. Extension metadata is preserved without interpreting it.

    Field::equal

    source
    fn Field::equal(Field, Field) -> Bool

    Field::new

    source
    fn Field::new(name : String, data_type : DataType, nullable? : Bool, metadata? : Array[KeyValue], children? : Array[Field]) -> Field

    Field::not_equal

    source
    fn Field::not_equal(x : Field, y : Field) -> Bool

    Field::to_repr

    source

    FileIndex

    source
    pub struct FileIndex {
      schema : Schema
      batches : Array[MessageInfo]
      dictionaries : Array[MessageInfo]
    } derive(
    Debug    )

    FileIndex::num_batches

    source
    fn FileIndex::num_batches(self : FileIndex) -> Int

    FileIndex::num_dictionaries

    source
    fn FileIndex::num_dictionaries(self : FileIndex) -> Int

    FileIndex::to_repr

    source

    FileIndex::total_body_bytes

    source
    fn FileIndex::total_body_bytes(self : FileIndex) -> Int64

    FileReader

    source
    pub struct FileReader {
      schema : Schema
      // private fields
    }

    Footer-indexed access to record batches in an in-memory Arrow IPC file.

    FileReader::get_batch

    source
    fn FileReader::get_batch(self : FileReader, index : Int) -> RecordBatch raise ArrowError

    FileReader::is_empty

    source
    fn FileReader::is_empty(self : FileReader) -> Bool

    FileReader::new

    source
    fn FileReader::new(data : Bytes, limits? : ReadLimits) -> FileReader raise ArrowError

    FileReader::num_batches

    source
    fn FileReader::num_batches(self : FileReader) -> Int

    FileWriter

    source
    pub struct FileWriter {
      // private fields
    }

    Incremental Arrow IPC file writer. Completed batch chunks can be sent directly to a sink. The writer retains footer block indices and dictionaries.

    FileWriter::finish

    source
    fn FileWriter::finish(self : FileWriter) -> Bytes raise ArrowError

    Return EOS, footer, footer length and trailing magic.

    FileWriter::new

    source
    fn FileWriter::new(schema : Schema) -> FileWriter raise ArrowError

    FileWriter::start

    source
    fn FileWriter::start(self : FileWriter) -> Bytes raise ArrowError

    FileWriter::write_batch

    source
    fn FileWriter::write_batch(self : FileWriter, batch : RecordBatch) -> Bytes raise ArrowError

    Return the framed batch and any first-use dictionary messages.

    IncrementalReader

    source
    pub struct IncrementalReader {
      // private fields
    }

    Incremental Arrow IPC stream decoder. A complete frame is released after each push, so memory retained by the decoder is bounded by one frame plus its active dictionaries. Call finish after the final input chunk.

    IncrementalReader::finish

    source
    fn IncrementalReader::finish(self : IncrementalReader) -> Unit raise ArrowError

    Signal final EOF. Clean EOF at a message boundary is allowed without EOS.

    IncrementalReader::new

    source

    IncrementalReader::num_batches

    source
    fn IncrementalReader::num_batches(self : IncrementalReader) -> Int

    IncrementalReader::push

    source
    fn IncrementalReader::push(self : IncrementalReader, chunk : Bytes) -> Array[RecordBatch] raise ArrowError

    Feed any byte chunk, including an empty one. Complete batches are returned in stream order. An error makes this reader terminal.

    IncrementalReader::schema

    source

    Returns a defensive schema copy as soon as a complete Schema frame arrives.

    IncrementalWriter

    source
    pub struct IncrementalWriter {
      // private fields
    }

    Incremental writer for the Arrow IPC stream format. Its output chunks can be written to a file or network sink without buffering completed batches.

    IncrementalWriter::finish

    source
    fn IncrementalWriter::finish(self : IncrementalWriter) -> Bytes raise ArrowError

    IncrementalWriter::new

    source

    IncrementalWriter::start

    source
    fn IncrementalWriter::start(self : IncrementalWriter) -> Bytes raise ArrowError

    IncrementalWriter::write_batch

    source
    fn IncrementalWriter::write_batch(self : IncrementalWriter, batch : RecordBatch) -> Bytes raise ArrowError

    Return one framed batch and any preceding dictionary update messages.

    IpcData

    source
    pub struct IpcData {
      schema : Schema
      batches : Array[RecordBatch]
    } derive(
    Debug    )

    Materialized IPC data. For batch-at-a-time decoding use StreamReader/FileReader.

    IpcData::to_repr

    source

    IpcData::to_table

    source
    fn IpcData::to_table(self : IpcData) -> Table raise ArrowError

    Convert a materialized IPC result into an owned Table.

    KeyValue

    source
    pub(all) struct KeyValue {
      key : String
      value : String?
    } derive(Eq, 
    Debug    )

    UTF-8 key/value metadata. Order and duplicate keys are preserved.

    KeyValue::equal

    source
    fn KeyValue::equal(KeyValue, KeyValue) -> Bool

    KeyValue::not_equal

    source
    fn KeyValue::not_equal(x : KeyValue, y : KeyValue) -> Bool

    KeyValue::to_repr

    source

    MessageInfo

    source
    pub(all) struct MessageInfo {
      offset : Int
      metadata_bytes : Int
      body_bytes : Int
      kind : MessageKind
      rows : Int?
      dictionary_id : Int64?
    } derive(
    Debug    )

    Metadata-only description of a framed IPC message. Inspectors validate framing and metadata bounds, but do not decode every record batch body.

    MessageInfo::to_repr

    source

    MessageKind

    source
    pub(all) enum MessageKind {
      SchemaMessage
      DictionaryMessage
      BatchMessage
    } derive(Eq, 
    Debug    )

    MessageKind::equal

    source
    fn MessageKind::equal(MessageKind, MessageKind) -> Bool

    MessageKind::not_equal

    source
    fn MessageKind::not_equal(x : MessageKind, y : MessageKind) -> Bool

    MessageKind::to_repr

    source

    Predicate

    source
    pub(all) enum Predicate {
      Eq(Int, Value)
      Ne(Int, Value)
      IsNull(Int)
      IsNotNull(Int)
      And(Predicate, Predicate)
      Or(Predicate, Predicate)
      Not(Predicate)
    } derive(
    Debug    )

    A small, typed predicate for batch and table filtering. Equality uses floating-point bit patterns, so -0.0 differs from +0.0 and equal NaN payloads compare equal. Null values never satisfy Eq or Ne.

    Predicate::to_repr

    source

    ReadLimits

    source
    pub(all) struct ReadLimits {
      max_rows_per_batch : Int
      max_values_per_batch : Int
      max_fields : Int
      max_batches : Int
      max_metadata_bytes : Int
      max_body_bytes : Int
      max_depth : Int
      max_dictionary_values : Int
      max_dictionary_messages : Int
    } derive(
    Debug    )

    Limits are checked before allocating decoded arrays. Input bytes are caller-owned.

    ReadLimits::default

    source
    fn ReadLimits::default() -> ReadLimits

    ReadLimits::to_repr

    source

    RecordBatch

    source
    pub struct RecordBatch {
      schema : Schema
      columns : Array[Column]
      num_rows : Int
    } derive(
    Debug    )

    A batch of equal-length columns. Writers revalidate mutable column arrays.

    RecordBatch::concat

    source
    fn RecordBatch::concat(self : RecordBatch, others : Array[RecordBatch]) -> RecordBatch raise ArrowError

    Append batches with exactly equal schemas. The result owns new arrays.

    RecordBatch::drop_column

    source
    fn RecordBatch::drop_column(self : RecordBatch, index : Int) -> RecordBatch raise ArrowError

    RecordBatch::filter

    source
    fn RecordBatch::filter(self : RecordBatch, mask : Array[Bool?]) -> RecordBatch raise ArrowError

    Keep rows whose mask is Some(true). None is treated as false.

    RecordBatch::filter_where

    source
    fn RecordBatch::filter_where(self : RecordBatch, predicate : Predicate) -> RecordBatch raise ArrowError

    Filter rows using a checked expression, preserving schema and row order.

    RecordBatch::get_row

    source
    fn RecordBatch::get_row(self : RecordBatch, index : Int) -> Array[Value?] raise ArrowError

    RecordBatch::head

    source
    fn RecordBatch::head(self : RecordBatch, count : Int) -> RecordBatch raise ArrowError

    RecordBatch::insert_column

    source
    fn RecordBatch::insert_column(self : RecordBatch, index : Int, field : Field, column : Column) -> RecordBatch raise ArrowError

    Insert one field and column at index, including at the end of a batch.

    RecordBatch::is_empty

    source
    fn RecordBatch::is_empty(self : RecordBatch) -> Bool

    RecordBatch::new

    source
    fn RecordBatch::new(schema : Schema, columns : Array[Column], num_rows? : Int) -> RecordBatch raise ArrowError

    num_rows is required only to represent nonzero rows with zero columns.

    RecordBatch::partition_where

    source
    fn RecordBatch::partition_where(self : RecordBatch, predicate : Predicate) -> (RecordBatch, RecordBatch) raise ArrowError

    Split a batch into matching and nonmatching rows, preserving row order in both results. Null values use the predicate's documented semantics.

    RecordBatch::project

    source
    fn RecordBatch::project(self : RecordBatch, indices : Array[Int]) -> RecordBatch raise ArrowError

    Select fields by index. Duplicate indices are kept in the requested order. The result has fresh mutable column, field and metadata arrays.

    RecordBatch::rename

    source
    fn RecordBatch::rename(self : RecordBatch, index : Int, name : String) -> RecordBatch raise ArrowError

    Rename a field without changing its data type, metadata or values.

    RecordBatch::replace_column

    source
    fn RecordBatch::replace_column(self : RecordBatch, index : Int, field : Field, column : Column) -> RecordBatch raise ArrowError

    Replace a column and its field together, keeping the same row count.

    RecordBatch::select

    source
    fn RecordBatch::select(self : RecordBatch, names : Array[String]) -> RecordBatch raise ArrowError

    Select named fields in request order. Duplicate input names are retained.

    RecordBatch::slice

    source
    fn RecordBatch::slice(self : RecordBatch, start : Int, length : Int) -> RecordBatch raise ArrowError

    Copy a contiguous range of rows. A zero-length slice at the end is valid.

    RecordBatch::split_rows

    source
    fn RecordBatch::split_rows(self : RecordBatch, max_rows : Int) -> Array[RecordBatch] raise ArrowError

    Split one batch into smaller batches; a zero-row batch remains one batch.

    RecordBatch::summaries

    source
    fn RecordBatch::summaries(self : RecordBatch) -> Array[ColumnSummary] raise ArrowError

    RecordBatch::tail

    source
    fn RecordBatch::tail(self : RecordBatch, count : Int) -> RecordBatch raise ArrowError

    RecordBatch::take

    source
    fn RecordBatch::take(self : RecordBatch, indices : Array[Int]) -> RecordBatch raise ArrowError

    Take rows in arbitrary order. Repeated indices are preserved.

    RecordBatch::to_repr

    source

    RecordBatch::validate

    source
    fn RecordBatch::validate(self : RecordBatch) -> Unit raise ArrowError

    Schema

    source
    pub(all) struct Schema {
      fields : Array[Field]
      metadata : Array[KeyValue]
    } derive(Eq, 
    Debug    )

    A schema, including ordered custom metadata.

    Schema::drop_field

    source
    fn Schema::drop_field(self : Schema, index : Int) -> Schema raise ArrowError

    Schema::equal

    source
    fn Schema::equal(Schema, Schema) -> Bool

    Schema::field_index

    source
    fn Schema::field_index(self : Schema, name : String) -> Int raise ArrowError

    Find one field by name. Duplicate names are ambiguous and are rejected.

    Schema::new

    source
    fn Schema::new(fields : Array[Field], metadata? : Array[KeyValue]) -> Schema

    Schema::not_equal

    source
    fn Schema::not_equal(x : Schema, y : Schema) -> Bool

    Schema::num_fields

    source
    fn Schema::num_fields(self : Schema) -> Int

    Schema::project

    source
    fn Schema::project(self : Schema, indices : Array[Int]) -> Schema raise ArrowError

    Select schema fields by index while copying field and metadata arrays.

    Schema::rename

    source
    fn Schema::rename(self : Schema, index : Int, name : String) -> Schema raise ArrowError

    Schema::to_repr

    source

    StreamIndex

    source
    pub struct StreamIndex {
      schema : Schema
      messages : Array[MessageInfo]
      has_eos : Bool
    } derive(
    Debug    )

    StreamIndex::num_batches

    source
    fn StreamIndex::num_batches(self : StreamIndex) -> Int

    StreamIndex::num_dictionaries

    source
    fn StreamIndex::num_dictionaries(self : StreamIndex) -> Int

    StreamIndex::to_repr

    source

    StreamIndex::total_body_bytes

    source
    fn StreamIndex::total_body_bytes(self : StreamIndex) -> Int64

    StreamReader

    source
    pub struct StreamReader {
      schema : Schema
      // private fields
    }

    Decode batches on demand from an in-memory stream. This retains the input Bytes; it is not a network push parser. Both explicit EOS and clean EOF work.

    StreamReader::new

    source
    fn StreamReader::new(data : Bytes, limits? : ReadLimits) -> StreamReader raise ArrowError

    StreamReader::next

    source
    fn StreamReader::next(self : StreamReader) -> RecordBatch? raise ArrowError

    Return the next batch. On error the reader becomes terminal.

    Table

    source
    pub struct Table {
      schema : Schema
      // private fields
    }

    A validated collection of record batches with one shared schema. Tables own snapshots of their input batches; mutating a caller array after new or append cannot change the stored rows.

    Table::append

    source
    fn Table::append(self : Table, batch : RecordBatch) -> Unit raise ArrowError

    Append one batch after validating the whole batch. The stored rows are detached from the caller's mutable arrays.

    Table::concat

    source
    fn Table::concat(self : Table, other : Table) -> Table raise ArrowError

    Concatenate tables with exactly equal schemas and retain batch boundaries.

    Table::drop_column

    source
    fn Table::drop_column(self : Table, index : Int) -> Table raise ArrowError

    Table::filter

    source
    fn Table::filter(self : Table, mask : Array[Bool?]) -> Table raise ArrowError

    Apply one nullable row mask over all batches. Null mask entries are false.

    Table::filter_where

    source
    fn Table::filter_where(self : Table, predicate : Predicate) -> Table raise ArrowError

    Table::from_file

    source
    fn Table::from_file(bytes : Bytes, limits? : ReadLimits) -> Table raise ArrowError

    Table::from_stream

    source
    fn Table::from_stream(bytes : Bytes, limits? : ReadLimits) -> Table raise ArrowError

    Table::get_batch

    source
    fn Table::get_batch(self : Table, index : Int) -> RecordBatch raise ArrowError

    Table::get_row

    source
    fn Table::get_row(self : Table, index : Int) -> Array[Value?] raise ArrowError

    Table::head

    source
    fn Table::head(self : Table, count : Int) -> Table raise ArrowError

    Table::is_empty

    source
    fn Table::is_empty(self : Table) -> Bool

    Table::new

    source
    fn Table::new(schema : Schema, batches : Array[RecordBatch]) -> Table raise ArrowError

    Table::null_counts

    source
    fn Table::null_counts(self : Table) -> Array[Int]

    Count null slots in each table column without constructing row scalars.

    Table::num_batches

    source
    fn Table::num_batches(self : Table) -> Int

    Table::num_rows

    source
    fn Table::num_rows(self : Table) -> Int

    Table::partition_where

    source
    fn Table::partition_where(self : Table, predicate : Predicate) -> (Table, Table) raise ArrowError

    Table::project

    source
    fn Table::project(self : Table, indices : Array[Int]) -> Table raise ArrowError

    Select columns in every batch, preserving batch boundaries and metadata.

    Table::rechunk

    source
    fn Table::rechunk(self : Table, max_rows : Int) -> Table raise ArrowError

    Rechunk batches to at most max_rows rows. This is useful for controlling downstream IPC message size, especially when source batches are uneven.

    Table::rename

    source
    fn Table::rename(self : Table, index : Int, name : String) -> Table raise ArrowError

    Table::select

    source
    fn Table::select(self : Table, names : Array[String]) -> Table raise ArrowError

    Table::slice_rows

    source
    fn Table::slice_rows(self : Table, start : Int, length : Int) -> Table raise ArrowError

    Return a range of global table rows. A range may cross batch boundaries.

    Table::summaries

    source
    fn Table::summaries(self : Table) -> Array[ColumnSummary] raise ArrowError

    Summarize all batches without combining or copying their column arrays.

    Table::tail

    source
    fn Table::tail(self : Table, count : Int) -> Table raise ArrowError

    Table::take_rows

    source
    fn Table::take_rows(self : Table, indices : Array[Int]) -> Table raise ArrowError

    Select global table rows in arbitrary order. Duplicate indices are kept. The output is a Table because source rows may come from separate batches.

    Table::to_file

    source
    fn Table::to_file(self : Table) -> Bytes raise ArrowError

    Table::to_ipc_data

    source
    fn Table::to_ipc_data(self : Table) -> IpcData raise ArrowError

    Return independent batch snapshots for callers using the original IPC API.

    Table::to_stream

    source
    fn Table::to_stream(self : Table) -> Bytes raise ArrowError

    TimeUnit

    source
    pub(all) enum TimeUnit {
      Second
      Millisecond
      Microsecond
      Nanosecond
    } derive(Eq, 
    Debug    )

    Resolution of a timestamp or duration. No timezone conversion is implied.

    TimeUnit::equal

    source
    fn TimeUnit::equal(TimeUnit, TimeUnit) -> Bool

    TimeUnit::not_equal

    source
    fn TimeUnit::not_equal(x : TimeUnit, y : TimeUnit) -> Bool

    TimeUnit::to_repr

    source

    Value

    source
    pub(all) enum Value {
      BooleanValue(Bool)
      Int8Value(Int)
      Int16Value(Int)
      Int32Value(Int)
      Int64Value(Int64)
      UInt8Value(Int)
      UInt16Value(Int)
      UInt32Value(UInt)
      UInt64Value(UInt64)
      Float32Value(Float)
      Float64Value(Double)
      Date32Value(Int)
      Date64Value(Int64)
      TimestampValue(Int64)
      DurationValue(Int64)
      Utf8Value(String)
      BinaryValue(Bytes)
      LargeUtf8Value(String)
      LargeBinaryValue(Bytes)
      FixedBinaryValue(Bytes)
      ListValue(Array[Value?])
      LargeListValue(Array[Value?])
      FixedListValue(Array[Value?])
      StructValue(Array[Value?])
      MapValue(Array[(Value, Value?)])
    } derive(
    Debug    )

    A typed value for row-wise construction. None represents an Arrow null.

    Value::to_repr

    source

    inspect_file

    source
    fn inspect_file(bytes : Bytes, limits? : ReadLimits) -> FileIndex raise ArrowError

    Inspect indexed Arrow file blocks without decoding record batch values.

    inspect_stream

    source
    fn inspect_stream(bytes : Bytes, limits? : ReadLimits) -> StreamIndex raise ArrowError

    Scan a stream's message framing without materializing its batches.

    read_file

    source
    fn read_file(data : Bytes, limits? : ReadLimits) -> IpcData raise ArrowError

    read_stream

    source
    fn read_stream(data : Bytes, limits? : ReadLimits) -> IpcData raise ArrowError

    Read and materialize all batches of an IPC stream.

    write_file

    source
    fn write_file(schema : Schema, batches : Array[RecordBatch]) -> Bytes raise ArrowError

    Write an uncompressed Arrow IPC file with footer-indexed record batches. This supports the implemented type subset of uncompressed Feather V2.

    write_stream

    source
    fn write_stream(schema : Schema, batches : Array[RecordBatch]) -> Bytes raise ArrowError

    Write an uncompressed V5 IPC stream, including a schema and end marker.

    Source Files

    Powered by MoonBit

    Site sourceReport issuePackagesBuild queueSkillsStatistics

    © 2026 mooncakes.io