Sign in

    MoonFrame

    A lightweight DataFrame and tabular-data library for MoonBit

    dataframe
    data-analysis
    csv
    tabular
    Download zip
    Author
    Version
    0.7.0
    License
    Apache-2.0
    Last updated
    14 days ago
    Downloads
    90

    #MoonFrame

    A small, friendly DataFrame library for MoonBit. Read a CSV, reshape it with a few chained methods, and print or export the result. If you have used pandas or polars, the shape of the API will feel familiar:

    // API shape (illustrative — the "Quick start" below runs this same
    // pipeline as a doc test)
    read_csv("sales.csv")
    .filter(col("product").eq(lit_str("widget")))
    .group_by([col("region")])
    .agg([col("revenue").sum()])
    .to_markdown()

    It covers CSV / JSON / NDJSON I/O, filtering, sorting, null handling, group-by, joins, summary statistics, a composable expression engine, and a lazy query layer, and exports to Markdown, HTML, JSON, NDJSON, and Vega-Lite charts — a focused foundation for everyday tabular work, not a full pandas clone.

    #Install

    MoonFrame is published on mooncakes.io. Add it to your module's dependencies:

    moon add ihb2032/MoonFrame

    Then import it with the @moonframe alias in the moon.pkg of the package that uses it:

    ///|
    import {
    "ihb2032/MoonFrame" @moonframe,
    }

    Now @moonframe.read_csv, the DataFrame / Series types, and every operator method are available in that package.

    MoonBit v0.10.4 deprecates the legacy JSON package manifest. New and migrated projects should use moon.mod / moon.pkg, as this repository does.

    #Compatibility

    The facade package — what import "ihb2032/MoonFrame" @moonframe re-exports — is the supported surface; the public sub-packages (@types, @series, @expr, @frame, @io, @chart, @lazy) stay directly importable for a slice of it. Two families are deliberately off the facade: the fluent-chain intermediates (WhenThen / GroupedDataFrame / … — chain through them without naming them), and the string-level serialisers, reached through @io by name.

    Pre-1.0, additions and fixes ride a patch version and a change to that surface rides the minor one. One case reads as additive but is not: adding a variant to a pub(all) enum (DataType, Scalar, DataError, …) is source-breaking under MoonBit's exhaustive match — only a caller whose match carries a wildcard arm stays compatible. The two error-detail enums (TypeMismatchDetail, ParseErrorDetail) are the exception: they are #non_exhaustive, so a match over them ends in a .. arm and a new diagnostic shape is a recompile, not a break. The upgrade steps for each release are in docs/migration.md.

    #Quick start

    Suppose you have a sales.csv:

    region,product,revenue,quantity west,widget,100,10 east,gadget,50,5 west,gadget,70,7 east,widget,30,3 north,widget,40,4 north,gadget,60,6 west,gizmo,90,9 east,gizmo,20,2

    Keep the widget rows, pick a few columns, and sort by quantity. The block below is a doc test: moon test compiles and runs it, so the example cannot drift from the API. It stages sales.csv itself through the public write_csv (the test suites stage fixtures the same way), uses the names the facade re-exports, and expects the rendered table inline — in your own module you write the same pipeline with the @moonframe. prefix from the import above:

    ///|
    test "readme: keep the widget rows, sort by quantity" {
    let path = "_build/readme_sales.csv"
    write_csv(
    path,
    DataFrame::DataFrame([
    Series::from_strings("region", [
    "west", "east", "west", "east", "north", "north", "west", "east",
    ]),
    Series::from_strings("product", [
    "widget", "gadget", "gadget", "widget", "widget", "gadget", "gizmo", "gizmo",
    ]),
    Series::from_ints("revenue", [100, 50, 70, 30, 40, 60, 90, 20]),
    Series::from_ints("quantity", [10, 5, 7, 3, 4, 6, 9, 2]),
    ]),
    )
    fn widgets(p : String) -> String raise DataError {
    read_csv(p)
    .filter(col("product").eq(lit_str("widget")))
    .select(cols(["region", "revenue", "quantity"]))
    // each key is (key, descending, nulls_last)
    .sort([(col("quantity"), true, true)])
    .to_markdown()
    }
    inspect(
    widgets(path),
    content=(
    #|| region | revenue | quantity |
    #|| ------ | ------- | -------- |
    #|| west | 100 | 10 |
    #|| north | 40 | 4 |
    #|| east | 30 | 3 |
    #|
    ),
    )
    }

    Every transformation is a method on DataFrame, so pipelines read top-to-bottom; anything that can fail raises DataError rather than crashing (see Error handling). For a fuller tour — group-by, joins, round-trips — see quickstart.mbt.md, which runs the same way.

    #What you can do

    • Read & write CSV, JSON, and NDJSON — read_csv / read_json / read_ndjson and their write_* counterparts, with tunable type inference, opt-in strict CSV quote validation, and formula neutralisation for spreadsheet-facing exports.
    • Reshape — filter, select, drop, rename, with_columns, multi-key sort, row dedup (unique), and null handling (drop_nulls, fill_null). Brackets read Polars-style: df["qty"] a column, df[1:3] / df[:-1] a row window, s[0] a cell.
    • Group & aggregate — group_by(keys).agg([...]) with sum / mean / min / max / count / std / variance / median / n_unique / first / last.
    • Express — composable column expressions (col("revenue") - col("cost"), & / | logic, when / then / otherwise, a str_* string namespace) feed with_columns / filter / agg, including compound reductions like (col("revenue") - col("cost")).sum(); map_elements / map_many drop to a host closure for anything past the built-in algebra.
    • Defer & optimize — LazyFrame::LazyFrame(df), or scan_csv / scan_ndjson for a lazy file source (deferred execution with projection and predicate pushdown into the reader, not streaming — the file still tokenizes at collect()), builds a query plan you can explain(); collect() runs it through the optimizer, producing an equal frame for the cells it reads. What a push-down does not read, it does not parse — so a parse error confined to a pruned column, or to a row the pushed-down predicate drops (in a column that predicate does not itself read), never surfaces. LazyFrame::collect's docstring states the contract in full.
    • Join — the full inner / left / right / outer / cross matrix on expression keys, e.g. orders.join(customers, JoinOptions::on([col("customer_id")])) — or, for differently-named or derived keys, JoinOptions::left_on([col("customer_id")], right_on=[col("id")]).
    • Summarize — describe() for a per-column summary, or single statistics (sum / mean / min / max / …).
    • Export — to_markdown() / to_html(), format_vega_lite (a Vega-Lite v5 chart spec), and — through the io package — the string-level format_csv / format_json / format_ndjson.

    For example, summarise the same data by region (another doc test, in-memory this time — group_by starts from any frame):

    ///|
    test "readme: summarise by region" {
    let sales = DataFrame::DataFrame([
    Series::from_strings("region", [
    "west", "east", "west", "east", "north", "north", "west", "east",
    ]),
    Series::from_ints("revenue", [100, 50, 70, 30, 40, 60, 90, 20]),
    Series::from_ints("quantity", [10, 5, 7, 3, 4, 6, 9, 2]),
    ])
    let summary = sales
    .group_by([col("region")])
    .agg([
    col("revenue").sum().with_alias("revenue"),
    col("quantity").sum().with_alias("quantity"),
    ])
    inspect(
    summary.to_markdown(),
    content=(
    #|| region | revenue | quantity |
    #|| ------ | ------- | -------- |
    #|| west | 260 | 26 |
    #|| east | 100 | 10 |
    #|| north | 100 | 10 |
    #|
    ),
    )
    }

    The same frame also exports as a styled HTML <table> via summary.to_html(options=HtmlOptions::HtmlOptions(caption="Summary")), or as a Vega-Lite v5 chart spec via format_vega_lite(summary, ChartSpec::bar("region", "revenue")) — ready to paste into the Vega editor.

    #Error handling

    Anything that can fail on bad input or I/O raises DataError; the library never aborts your program on a recoverable error. Call such functions inside a raise context (as the examples above do), or bridge back to a Result with a catch that re-wraps the error:

    ///|
    test "readme: bridging a raise back to a Result" {
    let df = DataFrame::DataFrame([Series::from_ints("a", [1, 2])])
    let result : Result[DataFrame, DataError] = Ok(df.select([col("missing")])) catch {
    e => Err(e)
    }
    assert_true(result is Err(ColumnNotFound(_)))
    }

    Operations that are provably total (head, to_markdown, …) just return their value. DataError is a pub(all) suberror, so you can match its variants (ColumnNotFound, ParseError, …) on the Err.

    #Documentation

    • The generated API reference — built from the docstrings, deployed to GitHub Pages on every merge to main, and published on mooncakes.io; one source, two hosts. Per-symbol contracts live in the docstrings, and that reference is where they are read.
    • quickstart.mbt.md — a runnable tour; every snippet and its expected output is executed by moon test, and by CI across all four backends, so a code block cannot drift from the API. The prose around them is reviewed, not executed
    • docs/comparison.md — how MoonFrame aligns with, and deliberately differs from, Polars / pandas
    • docs/performance.md — columnar layout, the Numeric fast path, and per-operation complexity
    • docs/type-inference.md — how CSV / JSON / NDJSON columns get their dtypes
    • docs/migration.md — upgrading across breaking releases
    • The release notes — what each release changed, written when it is cut

    Four runnable end-to-end programs live in examples/:

    moon run examples/sales_analysis # filter → select → sort → describe → markdown moon run examples/data_cleaning # drop_nulls → fill_null → CSV round-trip moon run examples/reporting # group_by → to_html + Vega-Lite spec moon run examples/expressions # with_columns → filter → agg → lazy + explain

    #Design notes

    MoonFrame's API and column semantics are modeled on Polars — see docs/comparison.md for the full alignment and the deliberate differences, and docs/performance.md for the columnar layout and per-operation complexity. A few things that surprise newcomers:

    • / is always Float (integer operands promote); dividing by zero gives IEEE ±inf / NaN, never a trap.
    • to_x never fails; as_x raises. Scalar::to_string renders any cell for display (a null cell gives ""); Scalar::as_string is a typed read that raises TypeMismatch on a wrong dtype. The wrong one does not fail — it silently stringifies — so reach for as_* when a wrong dtype is a bug you want reported, and to_* only for display text whatever the cell holds.
    • null and NaN are different. null is missing and propagates; NaN is a value (sum / mean / median propagate it, min / max skip it), and sort orders it by the IEEE total order — larger than every other value, so it trails an ascending sort and leads a descending one.
    • Comparisons are methods (col("a").gt(lit_int(0))), not >, and & / | are Kleene-logical, not bitwise — both are MoonBit constraints.

    #Contributing

    The codebase is a small, layered stack of packages; each has its own sources and a pkg.generated.mbti interface snapshot. Which kind of test a thing gets follows from what is under test rather than from which directory it sits in: a contract a caller can reach is tested from outside, through *_test.mbt, and a representation is tested from within, through *_wbtest.mbt, so that asserting it does not require making it public. Public packages are therefore mostly blackbox and internal/ ones mostly whitebox — an internal package with a contract of its own (a parser, a renderer, a comparison) has blackbox tests too, and one whose whole surface is driven from a single caller has none of its own (internal/kernel is covered through frame's expression evaluator, which is what exercises every kernel a caller can reach):

    types/ value types, errors (DataError), schemas internal/buffer/ the raw buffer primitives — the Arrow validity bitmap and the UTF-8 (bytes + offsets) string buffer every columnar layout flattens from internal/scan/ the scan-driver seam — what one readable source (CSV, NDJSON, a future Parquet) implements so the lazy engine executes, narrows, and renders it format-agnostically internal/column/ Arrow-style storage — validity bitmap + Builtin/Numeric backends; wrapped by Series and read by internal/kernel (which packages may name it is enforced by check_layering.sh) internal/kernel/ the vectorized expression kernels — Series broadcasting, arithmetic / logic / comparison / string ops, ternary, map, and the dtype inference behind a computed column; called by frame's evaluator internal/text/ shared text primitives — lexicographic compare, debug escaping, decimal literal parsing internal/numeric/ shared numeric primitives — exact Int64/Double comparison and the extremum fold, used from types up through the kernels internal/order/ shared position primitives — the stable index sort behind every sort, and the row-count clamp behind every head / tail / limit internal/ir/ module-internal expression AST — ExprNode + the operator tags, walked by the engine series/ Series + column-level stats + the shared reduction / rebuild / key-cell kernels expr/ opaque Expr handle — constructors, operators, when/then/otherwise builders, to_string rendering frame/ DataFrame + the operators (usually one per file) + group_by + join + the expression evaluator (with_columns / select / filter / agg) + to_markdown / to_html io/ CSV (NyaCSV-backed), JSON, NDJSON read / write + their options types chart/ Vega-Lite export — ChartSpec / ChartKind / VegaType builders, format_vega_lite / write_vega_lite (shares io's JSON cell conventions) lazy/ deferred query plan — LazyFrame builders, collect / explain, predicate + projection pushdown moonframe.mbt the root package — facade over the public API (fluent-chain intermediates stay in their sub-packages)

    The internal/ packages are MoonBit internal packages: importable inside this module only, so they carry no compatibility promise. Where a new piece of engine work belongs follows from what each layer owns:

    internal/column how a column is laid out: data buffers + validity bitmap series what a column is: dtype, validity, backend convergence internal/kernel how a column is computed: one vectorized pass per operator frame and above what a verb means: row sets, scheduling, schema, errors

    Two different relations are stacked there, and it helps to keep them apart: internal/kernel depends on series (it takes columns and hands columns back), while being its peer in storage access — both may name the physical column, because a vectorized pass needs the representation to keep the numeric fast paths. So a new vectorized operator goes in internal/kernel — where naming the physical column is the point — a new column-level primitive goes in series, and frame reads a column only through Series. frame's production build does not import internal/column at all; its test build does, to assert which backend an operator's output lands on.

    The dependency graph is a DAG, not a chain — expr and internal/ir sit off to one side of it. Which package may import which is not restated here on purpose: .github/scripts/check_layering.sh holds that rule and enforces it against the manifests, so there is one copy of it and it cannot quietly stop being true.

    The data model is an Apache Arrow-style column layout — a data buffer beside a byte-packed validity bitmap (1 = valid), except on the Numeric fast path, where an all-valid Int / Float column carries no bitmap at all — with an O(1) name→index cache; DataFrame::check_invariants() is a formal structural spec (INV1–INV5), and the operator test suites assert it over representative outputs. The usual loop:

    moon check # type-check the workspace moon test # run all tests (add --target all for every backend) moon fmt # format sources moon info # regenerate .mbti interface snapshots

    Contributions keep every source file fully covered (moon coverage analyze) and a warning-free moon check; CI also runs the moon bench suite, so keep it green too.

    #Acknowledgements

    MoonFrame is an original MoonBit implementation whose API and semantics are modeled on Polars (MIT) — the primary reference — with a few I/O conventions from pandas (BSD-3-Clause). No Polars or pandas source was translated; see docs/comparison.md for what is aligned, what deliberately differs, and what is out of scope.

    #License

    Apache-2.0 — see LICENSE.

    CellParseLocation

    Identifies how a typed cell's 1-based source position is reported. Row is used by CSV, Record by JSON arrays, and Line by NDJSON.

    ChartKind

    The mark type of a chart, mapped to a Vega-Lite top-level mark: Bar → "bar", Line → "line", Point → "point", Area → "area". pub(all) so callers can name the variants when building a ChartSpec (though the ChartSpec::bar / line / point / area constructors set the kind for you).

    ChartSpec

    A chart specification: the mark kind, the x / y encoding columns, an optional color grouping column, and an optional title. Fields are read-only outside the package — build a spec through one of the mark-named constructors (ChartSpec::bar(x, y) / line / point / area), naming color / color_type / title as needed:

    ChartSpec::bar("region", "revenue", title="Revenue by region")

    x / y / color are column names resolved against the frame at format_vega_lite time (a missing name raises ColumnNotFound); each column's dtype decides its Vega-Lite field type — the rule format_vega_lite's doc states (numeric → quantitative, Date → temporal, otherwise → nominal) — unless color_type overrides the color channel: e.g. a numeric grouping column (a cluster id) rendered as Nominal distinct colors rather than a continuous gradient (only color honours an override; x / y keep dtype inference).

    ClosedInterval

    Which endpoints an is_between range includes — Polars' closed. Both (the default) is lo <= x <= hi; Left / Right open the other end; None excludes both (lo < x < hi).

    CsvReadOptions

    Options that control how a CSV input is parsed into a DataFrame.

    • has_header — when true, the first row supplies column names. When false, every row is treated as data and column names are synthesised as "column1", "column2", … in declaration order; the first data row fixes the column count, so a later wider row's trailing cells are dropped under the lenient default (set strict_column_count = true to reject a ragged row instead).
    • delimiter — field separator passed through to NyaCSV.
    • infer_schema_rows — number of leading rows scanned when guessing each column's dtype. 0 (or any value <= 0) lifts the cap and scans every row (Polars' infer_schema_length=None), trading a slower inference pass for never mis-guessing a column from a prefix. Cells past a finite window are still parsed under the chosen dtype; one that doesn't fit is handled per on_parse_error.
    • null_values — raw strings that should be treated as null cells (both for type inference, which skips them, and for the final typed column, where they become None). The default [""] matches the empty string only, which is how an absent CSV cell is typically represented. Read it back through the null_values() accessor: the field itself is private, and both it and the constructor copy, so the token list a reader uses cannot be changed after the options are built.
    • strict_column_count — when true, every data row must have exactly as many cells as the header declares; a ragged row (too few or too many cells) raise ParseError instead of being silently null-padded (short row) or truncated (long row). Defaults to false, preserving the lenient behaviour that tolerates ragged rows (a permissive CSV idiom, and the reader's long-standing default).
    • on_parse_error — what to do when a non-null cell past the inference window fails to parse under its column's locked-in dtype. Raise (default) fails with ParseError(Cell(...)); Null downgrades the offending cell to a null cell and keeps the column's inferred dtype (Polars' ignore_errors=True).
    • allow_nonfinite_floats — when true (default), the float probe accepts every non-finite result: the nan / inf / infinity literals, and a finite literal that overflows Double to a signed Infinity (1e999), so a column of those tokens infers as Float. When false, the probe rejects them, so such a column falls back to String instead of being silently retyped to Float (a non-finite token past the window is then a parse failure handled per on_parse_error).
    • strict_quotes — when true, the input is pre-scanned and an unterminated quoted field, text after a closing quote, or a bare quote inside an unquoted field raise ParseError instead of being repaired by the lenient tokeniser. Defaults to false.

    CsvWriteOptions

    Options that control how a DataFrame is rendered as CSV text.

    • header — when true, the first emitted row is the column names; when false, only data rows are written.
    • delimiter — field separator written between cells.
    • null_value — the literal string written in place of a null cell. The default empty string round-trips with the reader's default null_values.
    • sanitize_formulas — when true, a String cell beginning with =, +, -, @, a tab, or a carriage return is prefixed with a single quote so a spreadsheet reads it as text. Deliberately lossy: such a cell reads back with the leading quote. Defaults to false.
    • line_terminator — the string written after every row, Polars' knob of the same name. The default "\n" (LF) matches every mainstream writer; RFC 4180's letter calls for "\r\n" (CRLF), and the reader accepts LF, CRLF, and CR alike, so either choice round-trips. Must not be empty — an empty terminator would fuse the rows into one.

    DataError

    Unified error type for all MoonFrame operations.

    The diagnostic shapes it carries — ParseErrorDetail, TypeMismatchDetail — are #non_exhaustive: a new shape is not a breaking change, and a caller matching those details carries a .. arm for the shapes it does not know. DataError itself stays exhaustive: #non_exhaustive is not supported on a suberror (MoonBit v0.10.14), so a new error kind remains a deliberate, breaking change under the enum-surface lock.

    DataFrame

    A column-oriented, schema-aware table. DataFrame owns an ordered list of equally-tall Series and a derived Schema; column lookup by name is O(1) through the schema's own index, which is the one name→position map a frame carries.

    The fields are priv (private to this package), so a frame can only ever be built through the constructors below — which rebuild the schema in lock-step with the column vector — and read through accessors that hand out nothing the caller can write back through: columns() and column_series() build a fresh array each call, and schema() returns a Schema, whose own fields are private behind a copying reader. External code cannot reach the live columns container, so it cannot mutate a validated frame into an inconsistent state.

    What every constructor here establishes and every transform preserves is that those parts agree. That agreement is written once — as INV1–INV5, checked by check_invariants — and deliberately not restated here, since a second copy is how the two would drift.

    DataType

    Logical data type for a column.

    Null is the dtype of a missing value, not of a column: it is what Scalar::Null.dtype() reports, and so what a TypeMismatch's "expected T, got Null" names when a null cell reaches a typed read. A hand-built Schema may also carry it, but nothing materialises a column on it — DataFrame::empty raises Unsupported for such a field, cast refuses it as a target, and no reader infers it: an all-null probe window falls back to String. Concrete columns always carry one of the other variants.

    A logical dtype is more than its variant: its display name, the physical buffer its columns ride, and its numericity are registered in the dtype registry, whose per-dtype rows are the compile-forced registration point for a dtype added here.

    Expr

    A composable column expression — the reified upgrade over the closure-based surfaces (the original row-predicate closure) and pre-materialised Series columns. An Expr is built through the constructor functions (col / lit / lit_* / when), the operator impls, and the methods, so every tree is well-formed by construction; the when chain stages through the WhenThen / WhenThenElse types below, completing as a Ternary. Building one is total — unknown columns and dtype mismatches surface at evaluation time (in frame), never here.

    The type is opaque: it wraps an ExprNode AST and exposes no variants. Outside this package an expression is a value you build and pass on, or render with to_string — there is no matching on its shape. The AST itself is module-internal, in a package a downstream module cannot import at all, so no caller can name a node, match one, or hold one: adding a node for a new operator breaks nobody.

    The shape is not observable either: there is no == on an expression. One existed, comparing the two trees, and it made how an operator lowers a promise to callers — normalising a tree or merging two node kinds would have changed what compared equal without changing what any expression means. to_string() is what a caller inspects with instead; it renders what it prints, so two literal series differing only in their cells render alike.

    The engine reads the shape through four accessors beside the type — node() hands over the wrapped AST, and children() / referenced_columns() / output_name() answer its introspection questions. frame and lazy are in-module and match ExprNode directly.

    The wrapped AST carries ExprNode children, not Expr, so this package bridges the two: a constructor unwraps its child Exprs with .node, and the children accessor below wraps ExprNode children back into Expr. The nodes, in construction-route order:
    • Col / Lit / LitSeries — leaves (col, lit / lit_*, lit_series); a LitSeries embeds a pre-materialised Series;
    • Binary — arithmetic + - * /, comparisons, Kleene & / |;
    • Unary — -e, .not(), the null / NaN probes, and .abs() / .floor() / .ceil() / .sign() / .round();
    • Agg — .sum() / .mean() / … reductions;
    • Str — the .str_* namespace;
    • Cast — .cast(dtype); Alias — .with_alias(name);
    • Ternary — when(c).then(a).otherwise(b);
    • FillNull / FillNan — .fill_null(v) / .fill_nan(v), dedicated nodes (not lowered to a Ternary) so the operand appears once and a chained coalesce stays linear;
    • IsIn / IsBetween — the membership / range predicates, likewise dedicated so the operand is evaluated once;
    • Map / MapBatches — the row-wise / batched closure escape hatches (.map_elements() / map_many() / .map_batches()), the function opaque so each is identified by its (label, inputs) (MapBatches also by its returns_scalar flag).

    Field

    Column metadata: a name, the column's logical DataType, and a declared nullable flag.

    nullable = false is a declared constraint that DataFrame::from_rows enforces: row data placing a Scalar::Null in such a column raises NullInNonNullable(name). (DataFrame::empty builds 0-row columns, so it can never violate the constraint.) The flag is never inferred from a column's contents: the constructor's nullable default — and so DataFrame::DataFrame and the IO readers — always sets nullable = true, so a nullable = false field only ever originates from an explicit Field::Field(..., nullable=false) in a caller-supplied schema.

    Once declared, it travels with the cells it describes. Every operation that moves a column carries its field rather than re-deriving one: Field::rename (and so DataFrame::rename / rename_with, which change nothing but the name), Schema::select / Schema::rename, the row-only frame transforms that reuse their input's schema verbatim (head / tail / slice / filter / sort / unique / fill_null / …), and the projections: a select or with_columns entry that is a bare col("x") (or an aliased one, which carries the field renamed), every column drop or select leaves in place, and the counter-prepending with_row_index. A column replaced through with_columns takes the field of whichever column now supplies its cells.

    An operation that computes cells derives a fresh field instead, with the constructor default: arithmetic, aggregations, cast, fill_null as an expression, group_by(...).agg(...), join (an outer one introduces nulls), and the summary frames (describe / null_count / sum / …). The declaration was made about the input's cells, not about these.

    That split is what keeps the flag honest without ever inspecting a column: it is validated once, by from_rows, and thereafter only accompanies cells it was validated against — the operations that carry it drop, reorder or rename, and never introduce a value.

    The fields are priv, so the struct is opaque outside this package: build one through Field::Field(...) and read it through name() / dtype() / nullable(). That is what actually keeps a future field additive — a readable field is also matchable, and MoonBit requires a struct pattern to name every field or carry .., so a public field's arrival would break a caller's pattern exactly as a new pub(all) enum variant breaks a match.

    HtmlOptions

    Options for DataFrame::to_html. Fields are read-only outside the package — build them through HtmlOptions::HtmlOptions(...), naming only what differs from the defaults:

    • max_rows — cap the number of data rows. None (the default) renders every row; Some(n) shows the first n and appends a <tfoot> banner (... (K more rows)) for the remainder. A negative n clamps to 0 (header + banner, no data rows), matching the Markdown renderer's row cap.
    • table_class — value of the <table class="..."> attribute. None (the default) omits the attribute.
    • caption — text of a leading <caption> element. None (the default) omits it.
    • escape — when true (the default) every emitted string — caption, header names, cell values, and the class attribute — has & / < / > / " / ' rewritten to their HTML entities, so untrusted data can't inject markup. Pass escape=false for trusted input that intentionally carries HTML.

    JoinOptions

    The knobs for DataFrame::join: which columns to match on, the join how, the suffix disambiguating a right column whose name collides with a left one, and whether to coalesce the key columns. Fields are read-only outside the package — build options through JoinOptions::on / JoinOptions::left_on / JoinOptions::cross, naming only the knobs that differ from their defaults. The three entry points are the legal shapes: an on join, a paired left_on / right_on join, and a keyless cross join — no combination of them can be spelled.

    • on — the key expressions, evaluated over both frames (Polars' IntoExpr). A bare col("id") matches on an existing column; a derived key such as col("ts") / lit_int(86400) matches on the computed value. ["region", "product"]-style multi-key joins become [col("region"), col("product")], matched on the tuple. An empty key list is invalid for on — a non-Cross join requires at least one key (DataFrame::join raises InvalidOperation); use JoinOptions::cross() for a Cartesian product.
    • left_on / right_on — paired key expressions evaluated on the left and right frame respectively, for joining on differently-named or differently-derived keys (left_on([col("a")]) against right_on([col("b")])). Mutually exclusive with on, and the two lists must name the same number of keys; key i on the left is matched against key i on the right.
    • how — Inner (the on / left_on default), Left, Right, Outer, or Cross.
    • suffix — appended to a right column whose name also occurs in the left frame (default "_right"); the left column keeps its name.
    • coalesce — whether each key column is merged into a single output column. None (the default) auto-selects by how, matching Polars: an inner / left / right join coalesces (the key appears once), while Outer does not (the right key is kept as <key><suffix>, null wherever its row had no match, so matched and unmatched rows are distinguishable). Some(true) always coalesces; Some(false) never does. Coalescing only applies when keying by on whose every key is a bare col(...) (the precondition for the key to name a single column on both sides); left_on / right_on and any derived key turn it off, keeping both key columns (Polars' "join on a non-column expression turns off coalescing"). When coalesced the single key takes its value from whichever side is present on that row (the two are equal on a matched pair) — the left on Inner / Left, the right on Right, and the present side per row on Outer. The three key lists are priv, read through the copying on_keys() / left_keys() / right_keys(). A public Array field is readable, and reading an array hands over the array itself — enough to push a key into an options value a LazyFrame::join already captured, changing what a built plan collects. The constructors copy on the way in for the same reason; these accessors close the way out. The remaining fields are immutable values, so they stay public.

    JoinType

    How an equi-join combines rows that share (or fail to share) a key.

    pub(all) so callers can name the variants directly (e.g. JoinOptions::on(keys, how=Left)):
    • Inner — keep only rows whose key matches on both sides.
    • Left — keep every left row; where it has no right match, the right-hand columns are filled with nulls.
    • Right — the mirror of Left: keep every right row; where it has no left match, the left-hand columns are filled with nulls. The output still leads with the left columns (only the row set changes to "all right rows").
    • Outer — the full outer join: keep matched pairs plus every unmatched row from both sides (the missing side's columns null). Left rows (matched and unmatched) come first in left order, then the unmatched right rows in right order.
    • Cross — the Cartesian product (every left row paired with every right row); takes no key columns.

    JsonReadOptions

    Options that control how a JSON records payload is parsed into a DataFrame.

    • infer_schema_rows — number of leading records inspected when guessing each column's dtype. 0 (or any value <= 0) lifts the cap and scans every record (Polars' infer_schema_length=None). Records past a finite window are still parsed under the chosen dtype; one that doesn't fit is handled per on_parse_error.
    • on_parse_error — what to do when a non-null cell past the inference window fails to parse under its column's locked-in dtype: Raise (default) fails with ParseError(Cell(...)); Null downgrades the offending cell to a null and continues (Polars' ignore_errors=True).
    • reject_duplicate_keys — whether an object repeating a key is an error. The default false keeps the serde / pandas / Polars behaviour: a repeated key survives only as its last value, the earlier ones silently dropped. true rejects the read instead — ParseError(Message(...)) naming the key and the 1-based object it repeated in — so a payload that would silently lose cells fails loudly. Nested objects are scanned too, and key spellings are compared decoded ("a" and "\u0061" are the same key). The scan runs before the parse and only when the option is on, so the default read pays nothing.

    The NDJSON reader shares this type: the two formats differ in framing, not in what there is to configure.

    KeepStrategy

    Which row of a set of duplicates DataFrame::unique keeps — Polars' keep parameter. Two rows are duplicates under the same composite KeyCell encoding group_by / join use (a Float NaN equals NaN, -0.0 folds into +0.0, and a null cell is an ordinary value).

    • First keeps the earliest occurrence of each distinct row;
    • Last keeps the latest;
    • None keeps only rows that have no duplicate at all — every row of a duplicated set is dropped (Polars' keep='none').

    The kept rows always come out in ascending original-row order, so the result is deterministic without a sort regardless of strategy.

    LazyFrame

    The public face of the lazy layer: LazyFrame wraps a LogicalPlan and grows it through builder methods that mirror the eager DataFrame verbs name-for-name. Building is total — every builder just wraps the plan in one more node, so a LazyFrame can always be constructed, chained, and explained, even when collecting it would fail. All computation (and all failure) happens in collect, which optimizes the plan and then interprets it through the public eager operators: LazyFrame::LazyFrame(df).f(…).g(…).collect() equals df.f(…).g(…) — the faithful-deferred-executor contract the package's tests pin, and the reason the optimizer's rewrites have to preserve results rather than merely preserve rows.

    Every builder that takes an array copies it, so a plan can never observe later mutation of a caller's argument — the immutability the paragraph above promises holds for the arguments too, not just the captured frames.

    The entry point is the type's own constructor, LazyFrame::LazyFrame(df) — not a DataFrame method, which would have to live in frame and close a frame ↔ lazy import cycle, and not lazy(df), since lazy is a MoonBit reserved word.

    OnParseError

    What a reader does when a non-null cell fails to parse under the dtype that inference locked in for its column — in practice a value past the inference window that doesn't fit (e.g. a String in a column inferred Int from its first rows).

    • Raise (every reader's default) — surface ParseError(Cell(...)), failing the whole read. The strict, lossless behaviour: a malformed cell is never silently dropped.
    • Null — downgrade the offending cell to a null cell and keep going, the resilient behaviour Polars exposes as ignore_errors=True (pandas tolerates the same). The column keeps its inferred dtype; only the unparseable cells become null.

    Shared by CsvReadOptions and JsonReadOptions and consumed in build_typed_column's parse-failure branch, so the three readers agree on the policy.

    ParseErrorDetail

    Structured details carried by DataError::ParseError. #non_exhaustive: a new diagnostic shape here is not a breaking change — a caller matching these details carries a .. arm for the shapes it does not know (the contract the DataError variants naming them inherit).

    Scalar

    A single cell value in a DataFrame.

    Null represents a missing value. The as_* conversions raiseTypeMismatch(Expected(expected, got, "")) on a wrong dtype (got = Null for a nullcell). Comparisons between cells are column-level — an expression'seq / lt / gt` verbs — and their dtype matrix lives on the comparison kernel, not here.

    The Float variant carries a 64-bit Double; the Int variant carries a 64-bit Int64. Use as_float for explicit access; Int is also accepted via numeric promotion to Double. The Date variant carries whole days since the Unix epoch (1970-01-01 = day 0), rendering as an ISO YYYY-MM-DD string.

    Schema

    Ordered list of Fields describing a DataFrame's columns.

    Construction always validates uniqueness of names; once built, a Schema is guaranteed to have no duplicate column names.

    The fields array is priv: build a schema with Schema::Schema and read a copy via fields() / field_names(). External code cannot reach or mutate the backing array, so a validated schema stays valid.

    index is the name→position map behind index_of, and through it field / select / rename. It is derived from fields and built in lock-step with it, never separately: the constructors are the only writers, and each one builds both. Uniqueness is what makes it total — every name maps to exactly the position that carries it — so it is the same validation pass that establishes both. A schema is immutable, so one scan at construction replaces a scan per lookup: resolving c names against a c-column schema was O(c²) while every resolution walked the array, which is the shape a wide select or rename hits.

    Series

    Series has its own package (extracted so the expression layer can build on the per-column unit); re-exporting the type carries its constructors (from_ints / …) and methods along, so the facade name for Series is unchanged for callers.

    TypeMismatchDetail

    Structured details carried by DataError::TypeMismatch. #non_exhaustive: a new diagnostic shape here is not a breaking change — a caller matching these details carries a .. arm for the shapes it does not know (the contract the DataError variants naming them inherit).

    VegaType

    A Vega-Lite field type, used to override the dtype-based inference for the color channel. pub(all) so callers can name the variants in a constructor's color_type. Quantitative is a continuous measure (a color gradient), Nominal an unordered category (distinct colors per value), Ordinal an ordered category, and Temporal a time field.

    col

    fn col(name : String) ->
    Expr

    Reference a column by name, for the Polars-style call shape col("a") + col("b"). The name resolves at evaluation time — ColumnNotFound then if the frame has no such column; building the reference itself is total. A free function, like the lit* family — the facade re-exports it.

    Example

    test "col: renders as a reference" {
    inspect(col("qty").to_string(), content="col(qty)")
    }

    cols

    fn cols(names : Array[String]) -> Array[
    Expr
    ]

    Build a column-reference list from names: cols(["a", "b"]) is exactly [col("a"), col("b")]. The ergonomic shorthand for the common case of projecting (or dropping) several existing columns by name through the expression verbs — df.select(cols(["a", "b"])) reads almost like the names-only projection it replaces. Each name resolves at evaluation time, exactly like col; building the list is total.

    Example

    test "cols: shorthand for the reference list" {
    debug_inspect(
    cols(["a", "b"]).map(e => e.to_string()),
    content="[\"col(a)\", \"col(b)\"]",
    )
    }

    format_vega_lite

    Render df and spec as a complete Vega-Lite v5 specification JSON string: $schema (pinned to the Vega-Lite v5 schema URL) + optional title + mark (from spec.kind) + encoding (x / y and, when set, color, each {field, type}) + data.values (the frame inlined as JSON records, sharing format_json's cell mapping — null and non-finite-float cells become JSON null). A frame with the encoded columns but zero rows yields "data":{"values":[]}.

    Each encoding channel's field type follows its column's dtype — the single statement of the dtype→type rule: a numeric column (Int / Float) is "quantitative", a Date column is "temporal" (its ISO cells — data.values' date units, via the JSON records' ISO mapping — sort chronologically), and every other dtype is "nominal" — unless the channel carries an explicit color_type override.

    The spec's x / y / color columns are resolved left-to-right; the first name absent from df raises ColumnNotFound(name) — this is the one reason the function is not total (it accepts column names, and a name can be wrong). The emitted text is always valid JSON (it is built through @json and stringified), so it round-trips through any standards-compliant JSON / Vega-Lite reader.

    lit

    Embed a literal Scalar. At evaluation it becomes a length-1 column broadcast against its siblings. Prefer the typed shorthands (lit_int / lit_float / lit_str / lit_bool), which avoid spelling the Scalar variant at call sites.

    Example

    test "lit: embeds any Scalar" {
    inspect(lit(@types.Scalar::Int(3L)).to_string(), content="3")
    }

    lit_bool

    fn lit_bool(value : Bool) ->
    Expr

    Bool literal.

    Example

    test "lit_bool: renders bare" {
    inspect(lit_bool(true).to_string(), content="true")
    }

    lit_float

    fn lit_float(value : Double) ->
    Expr

    Float literal.

    Example

    test "lit_float: whole values keep their point" {
    inspect(lit_float(2.0).to_string(), content="2.0")
    }

    lit_int

    fn lit_int(value : Int64) ->
    Expr

    Int literal (64-bit, like every MoonFrame integer cell).

    Example

    test "lit_int: renders bare" {
    inspect(lit_int(5L).to_string(), content="5")
    }

    lit_series

    Embed a pre-materialised Series as a literal column, so a ready-made column joins a pipeline beside the declarative col(...) ... expressions (df.with_columns([lit_series(s), (col("a") + col("b")).with_alias("c")])). At evaluation the series is used as-is: a length-1 series broadcasts over the evaluation height (like a scalar lit), a series whose length matches the frame supplies one cell per row, and any other length raises LengthMismatch. The result column keeps the series' own name unless with_alias overrides it — so with_columns([lit_series(s)]) adds, or in-place replaces, a column named s.name(). A free function (the argument is a Series, not an Expr), like the lit_* family and map_many. Building the node is total; the length check happens at evaluation, in frame.

    Example

    test "lit_series: names its column after the series" {
    let s = @series.Series::from_ints("score", [1, 2])
    inspect(lit_series(s).output_name(), content="score")
    }

    lit_str

    fn lit_str(value : String) ->
    Expr

    String literal.

    Example

    test "lit_str: renders quoted" {
    inspect(lit_str("ab").to_string(), content="\"ab\"")
    }

    map_many

    Apply a host closure across several input columns, row by row — the multi-input escape hatch, and the reified replacement for the original closure filter predicate: f is handed one Scalar per input in inputs order (null cells as Scalar::Null) and is called for every row, so a row predicate is a map_many(..., f) returning a Bool. A free function rather than a method (there is no single self); inputs may mix columns, literals, and aggregations, the length-1 results broadcasting over the row count. Same opacity, totality, dtype inference, naming (after the leftmost input), and value-barrier rules as map_elements — with the dtype fallback reading the leftmost input, whatever it is, so only an empty inputs leaves an all-null result with no witness and raises Unsupported. The inputs array is copied, so mutating it after construction cannot alter the built expression.

    Example

    test "map_many: drops to a host closure" {
    let total = map_many(label="total", [col("a"), col("b")], cells => {
    @types.Scalar::Int(cells[0].as_int() + cells[1].as_int())
    })
    inspect(total.to_string(), content="map(\"total\", [col(a), col(b)])")
    }

    read_csv

    Read a CSV file. options defaults to CsvReadOptions::CsvReadOptions(), so read_csv(path) is the default-options read. IOError from the file system surfaces as raise IoError(message). CsvReadOptions::CsvReadOptions(strict_quotes=true) rejects malformed quoting before NyaCSV tokenisation, exactly as parse_csv_str does.

    read_json

    Read a JSON records file. options defaults to JsonReadOptions::JsonReadOptions(), so read_json(path) is the default-options read. Filesystem errors surface as raise IoError(message).

    read_ndjson

    Read an NDJSON file. options defaults to JsonReadOptions::JsonReadOptions(), so read_ndjson(path) is the default-options read. Filesystem errors surface as raise IoError(message).

    scan_csv

    The lazy CSV source: scan_csv(path) builds a LazyFrame whose leaf is a deferred read of path — the push-down-aware counterpart of eager read_csv. Nothing is read or parsed until collect: the builder is total, like every other LazyFrame constructor, so a missing file or a malformed CSV only surfaces then (as the eager reader's IoError / ParseError / DuplicateColumn). It is not a streaming read — the reader still tokenises the whole file, and streaming it is tracked as future work.

    The payoff over LazyFrame::LazyFrame(read_csv(path)) is that this leaf absorbs both push-downs, so the work it skips is work the eager pipeline does:
    • projection — the optimizer narrows the leaf to the columns the pipeline provably consumes, and only those are parsed, so scan_csv("sales.csv").select([col("region"), col("revenue")]).collect() never builds the columns it drops;
    • predicate — a filter sitting on the leaf moves into the read: the reader builds the predicate's columns, asks which rows survive, then parses the remaining columns for the survivors alone. Only the first such filter is absorbed; a second stays a node above the scan.

    Both prune cells an eager read-then-filter would have parsed, which is the one way a narrowed scan diverges from a full eager read: a ParseError confined to a dropped column, or to a row the predicate drops in a column the predicate does not read, does not surface. Dtype inference still walks the whole file, so dtypes and the surviving cells match the eager read exactly.

    options (delimiter, header, null strings, inference window, parse-error policy, strict quotes, …) defaults to CsvReadOptions::CsvReadOptions(), mirroring eager read_csv. It is captured into the plan and applied by the reader at collect time; projection push-down narrows which columns those options parse, never which options apply. The projection starts None (read every column) — only the optimizer fills it.

    scan_ndjson

    The lazy NDJSON source: scan_ndjson(path) builds a LazyFrame whose leaf is a deferred read of path — the push-down-aware counterpart of eager read_ndjson, and the line-oriented sibling of scan_csv. Like every LazyFrame constructor it is total: nothing is read or parsed until collect, so a missing file or a malformed line only surfaces then (as the eager reader's IoError / ParseError). It is not a streaming read — the reader still parses every line — and it absorbs the same two push-downs scan_csv documents: projection (only the consumed columns are built) and the first predicate sitting on the leaf (the surviving rows' cells are the only ones built past the predicate's own columns). So a parse error confined to a dropped column, or to a dropped row, does not surface; the surviving cells and the inferred dtypes match an eager read exactly. (There is no scan_json for the single-array shape [{...}], which must be parsed whole to find its records — so nothing can be pruned at read time, the same reason Polars has scan_ndjson but no scan_json.)

    options (inference window, parse-error policy) defaults to JsonReadOptions::JsonReadOptions(), mirroring eager read_ndjson. It is captured into the plan and applied by the reader at collect time; projection push-down narrows which columns those options parse, never which options apply. The projection starts None (read every column) — only the optimizer fills it.

    when

    Open a conditional expression: when(cond).then(a).otherwise(b) evaluates to a where cond is true, b where it is false, and null where cond is null. The chain is the only construction route, so a conditional is complete by the time it becomes an Expr.

    Example

    test "when: the chain is the only construction route" {
    let tagged = when(col("qty").gt(lit_int(5)))
    .then(lit_str("big"))
    .otherwise(lit_str("small"))
    inspect(
    tagged.to_string(),
    content="when((col(qty) > 5)).then(\"big\").otherwise(\"small\")",
    )
    }

    write_csv

    Write a DataFrame to a CSV file. options defaults to CsvWriteOptions::CsvWriteOptions(), so write_csv(path, df) is the default-options write. IOError from the file system surfaces as raise IoError(message). A string cell (or column name) holding an unpaired UTF-16 surrogate is refused with raise InvalidOperation — the UTF-8 file encoding would swallow the following delimiter and silently shift every later cell boundary. CsvWriteOptions::CsvWriteOptions(sanitize_formulas=true) applies the same spreadsheet-safety transformation as format_csv before writing. Rendering runs through format_csv first, so its refusals come first too — an unrepresentable delimiter and the N×0 shape (rows but no columns) both raise InvalidOperation before the file is opened, leaving nothing written.

    write_json

    Write a DataFrame to a JSON records file. Filesystem errors surface as raise IoError(message). A string cell (or column name) holding an unpaired UTF-16 surrogate is refused with raiseInvalidOperation — the UTF-8 file encoding would swallow the following code unit and corrupt the JSON framing.

    write_ndjson

    Write a DataFrame to an NDJSON file. Filesystem errors surface as raise IoError(message). A string cell (or column name) holding an unpaired UTF-16 surrogate is refused with raise InvalidOperation — the UTF-8 file encoding would swallow the following code unit and corrupt the line framing.

    write_vega_lite

    Write a Vega-Lite v5 spec for df / spec to path. Mirrors write_json: a ColumnNotFound from format_vega_lite (a spec column absent from df) propagates unchanged, a filesystem failure surfaces as raise IoError(message), and content holding an unpaired UTF-16 surrogate is refused with raise InvalidOperation.

    Source Files