README- UCellParseLocation
- UChartKind
- UChartSpec
- UClosedInterval
- UCsvReadOptions
- UCsvWriteOptions
- UDataError
- UDataFrame
- UDataType
- UExpr
- UField
- UHtmlOptions
- UJoinOptions
- UJoinType
- UJsonReadOptions
- UKeepStrategy
- ULazyFrame
- UOnParseError
- UParseErrorDetail
- UPivotAgg
- UQuantileInterpolation
- URankMethod
- UScalar
- USchema
- USeries
- UTypeMismatchDetail
- UVegaType
- Vcol
- Vcols
- Vformat_vega_lite
- Vlit
- Vlit_bool
- Vlit_float
- Vlit_int
- Vlit_series
- Vlit_str
- Vmap_many
- Vread_csv
- Vread_json
- Vread_ndjson
- Vscan_csv
- Vscan_json
- Vscan_ndjson
- Vwhen
- Vwrite_csv
- Vwrite_json
- Vwrite_ndjson
- Vwrite_vega_lite
ihb2032/MoonFrame/lazy does not have a README file
LazyFrame
pub struct LazyFrame {
// private fields
}The public face of the lazy layer: LazyFrame wraps a LogicalPlan
and grows it through builder methods that mirror the eager DataFrame
verbs name-for-name. Building is total — every builder just wraps
the plan in one more node, so a LazyFrame can always be constructed,
chained, and explained, even when collecting it would fail. All
computation (and all failure) happens in collect, which optimizes the
plan and then interprets it through the public eager operators:
LazyFrame::LazyFrame(df).f(…).g(…).collect() equals df.f(…).g(…) — the
faithful-deferred-executor contract the package's tests pin, and
the reason the optimizer's rewrites have to preserve results rather than
merely preserve rows.
Every builder that takes an array copies it, so a plan can never
observe later mutation of a caller's argument — the immutability the
paragraph above promises holds for the arguments too, not just the
captured frames.
The entry point is the type's own constructor, LazyFrame::LazyFrame(df)
— not a DataFrame method, which would have to live in frame and close a
frame ↔ lazy import cycle, and not lazy(df), since lazy is a MoonBit
reserved word.
LazyFrame::LazyFrame
Wrap an in-memory frame as the leaf of a new plan (a Scan node):
LazyFrame::LazyFrame(df).filter(…).collect(). Total — the frame is
captured as-is, and frames are immutable, so the plan can never observe
later state.
LazyFrame::bottom_k
Defer DataFrame::bottom_k: the k rows smallest under by, reverse~
flipping the end — top_k's mirror with its own node for the same
collect-time-validation reason.
LazyFrame::collect
Run the plan and materialize the result — the only point in the lazy
layer that computes (or fails). The plan first passes through the
total optimizer rewrites (the optimizer's pass and rule tables are
the pipeline's inventory):
row-window nodes sink toward the scan — neighbouring windows fusing
into their composite where the composite provably keeps the eager
rows — filters sink below the
stages they provably commute with, so rows drop as early as possible,
and a required-columns pass then drops scan columns nothing downstream
reads (and, for a scan_csv source, never parses them); the rules
then fold column-free expression subtrees to literals and merge
adjacent with_columns stages that commute — unless the
plan is DAG-shaped (the same LazyFrame reaching both sides of a
join), in which case it runs as built: a shared subtree makes the
rewrite passes' node count explode, so they stand down rather than
trade soundness for the work. The rewrites'
contract is exactly this method's, so they change
what work happens, and for a successful result never what comes back: the
(optimized) plan is walked bottom-up, delegating each node to the public
eager operator it defers, so the result equals running the same
verbs eagerly in the same order. Errors match the eager operators'
(ColumnNotFound, TypeMismatch, IndexOutOfBounds, …) with one
carve-out: a scan_csv / scan_ndjson source absorbs projection and
predicate pushdown, so it never parses a column no consumer reads, nor the
non-predicate cells of a row the predicate drops — a ParseError confined
there, which an eager read-then-filter would raise, does not surface; and
in a file where several columns each hold a parse error, the pushed-down
read (which builds its predicate key columns first) can name a different
one than the eager read would. Every
other operator still produces its eager error. To see the rewritten plan
this method actually runs, render it with explain(optimized=true).
LazyFrame::count
Defer DataFrame::count: a 1-row Int frame of each column's non-null cell
count (every dtype).
LazyFrame::drop
Defer DataFrame::drop: remove the columns named by exprs (bare col
references), keeping every other column. A name absent from the frame, or a
non-col expression, surfaces as the eager drop's error at collect time.
LazyFrame::drop_nulls
Defer DataFrame::drop_nulls: drop every row with a null in subset (a
list of col references), or — with no subset — in any column. A name
absent from the frame surfaces at collect time.
LazyFrame::explain
Render the logical plan as an indented tree — the root operation on
the first line, inputs two spaces deeper, expressions in their
documented Show form, and SCAN [rows×cols] leaves:
SELECT [col(region), col(adj)]
WITH_COLUMNS [(col(revenue) * 1.1) as adj]
FILTER (col(region) == "west")
SCAN [4×3]By default this is the plan as built — a faithful mirror of the
chained verbs, which is the package's contract. Pass optimized=true
to render the plan collect actually runs instead: a sunk FILTER
appears below the stages it crossed — or disappears into the leaf as a
WHERE suffix, when the stage it reaches is a file source that can apply
it while reading — and the projection pushdown shows
up as a narrowing SELECT over an in-memory SCAN or as the column list
on a SCAN_CSV source, so printing both forms is the before/after view
of what the optimizer moved and pruned — except on a DAG-shaped plan
(the same LazyFrame on both sides of a join), which collect runs
as built, so optimized=true renders the same tree as the default
under a leading NOTE: shared subplan line saying the rewrites were
skipped (a shared subtree is also marked "(shared, rendered above)").
(Plans are immutable, so the flag rewrites a copy and never perturbs
this frame.)
Total either way — the rewrite is a pure tree walk, so a plan that
would fail to collect still explains, which is the point: inspect
first, compute later.
LazyFrame::fill_nan
Defer DataFrame::fill_nan: every Float column's NaN cells replaced
with value, every other column verbatim.
LazyFrame::fill_null
Defer DataFrame::fill_null: replace every null cell, in every column,
with value. A column whose dtype does not match value surfaces the
eager verb's error at collect time.
LazyFrame::fill_null_backward
Defer DataFrame::fill_null_backward: the from-below mirror of
fill_null_forward.
LazyFrame::fill_null_forward
Defer DataFrame::fill_null_forward: every column's nulls filled from
above, limit? capping the consecutive slots one value may fill. The
limit validation surfaces as the eager error at collect.
LazyFrame::filter
Defer DataFrame::filter: keep the rows where predicate
evaluates to true (false / null cells drop the row). Total at build
time — a predicate over missing columns or of a non-Bool type only
fails when the plan is collected.
LazyFrame::first
Defer the first row as a 1-row frame (an exclusively lazy verb — the
eager face reaches the same rows through head(1)). A 0-row input
collects to the 0-row frame: the slice window simply closes.
LazyFrame::gather_every
Defer DataFrame::gather_every: every n-th row from offset. The
stride validation (n >= 1, a non-negative offset) surfaces as the
eager error at collect, keeping the two layers' failures identical.
LazyFrame::group_by
Defer DataFrame::group_by: attach grouping key expressions to the plan
and return a LazyGroupBy waiting for its aggregations. Each key is an
Expr evaluated over the whole frame at collect time (a bare col, a
derived key, or a length-1 key broadcasting to one group), exactly like
the eager verb. Nothing is partitioned yet — like every builder this is
total, so a missing column, a dtype clash, or a repeated output name
only surfaces at collect time, as the eager group_by's
ColumnNotFound / TypeMismatch / DuplicateColumn. Only
LazyGroupBy::agg grows the plan; a LazyGroupBy is never itself
collectable, mirroring how the eager GroupedDataFrame is not a frame.
LazyFrame::head
Defer DataFrame::head: the first n rows. Inherits eager head's
total clamp — n beyond the frame keeps every row, a negative n
keeps every row but the last |n| — so collecting can't fail on this
node. Also exposed under its SQL name limit (via #alias): the same
deferred node, so lf.limit(n) collects to the same rows and
explains as HEAD n.
LazyFrame::join
Defer DataFrame::join: combine this plan's output (the left side)
with another plan's output (the right side) under options — each
side carries its own deferred pipeline. Key resolution, type checks,
and the cross-join key rules all surface at collect time.
LazyFrame::last
Defer the last row as a 1-row frame (an exclusively lazy verb, the twin
of first): the window from one past-the-end — a negative offset
counts from the end — so a 0-row input collects to the 0-row frame.
LazyFrame::max
Defer DataFrame::max: the maximum counterpart of min.
LazyFrame::mean
Defer DataFrame::mean: a 1-row frame of each numeric column's mean as
Float (empty / all-null → Null), non-numeric columns a Null cell.
LazyFrame::median
Defer DataFrame::median: a 1-row frame of each numeric column's median
as Float (NaN propagating, the two middles averaged on an even
count), non-numeric / empty / all-null columns a Null cell.
LazyFrame::min
Defer DataFrame::min: a 1-row frame of each numeric column's minimum
(NaN skipped, source dtype kept), non-numeric columns a Null cell.
LazyFrame::n_unique
Defer DataFrame::n_unique: a 1-row Int frame of each column's
distinct value count (every dtype; a null counts as one distinct value,
every NaN one bucket — the documented n_unique rule).
LazyFrame::null_count
Defer DataFrame::null_count: a 1-row Int frame of each column's null
count — the complement of count.
LazyFrame::pivot
Defer DataFrame::pivot: the long → wide reshape, one output row per
distinct index? combination and one output column per distinct on~
value, slots collapsed by aggregate_function? (omitted: a multi-cell
slot's InvalidOperation, at collect). The omission defaults resolve
against the collected frame, so collect is the eager result verbatim;
a derived key evaluates there too, exactly as the eager verb's.
LazyFrame::quantile
fn LazyFrame::quantile(self : LazyFrame, q : Double, interpolation? : QuantileInterpolation) -> LazyFrameDefer DataFrame::quantile: a 1-row frame of each numeric column's
q-th quantile under interpolation (default Nearest), with median's
Float-or-Null cell rules. A q outside [0, 1] surfaces as
InvalidOperation at collect.
LazyFrame::rename
Defer DataFrame::rename: apply each (from, to) rename in order. A
missing source name or a colliding target surfaces at collect time.
LazyFrame::rename_with
Defer DataFrame::rename_with: rename every column through f
(new = f(old)). A collision — two columns f maps to the same name —
surfaces as the eager rename_with's DuplicateColumn at collect time.
LazyFrame::reverse
Defer DataFrame::reverse: flip the row order at collect time. Total.
LazyFrame::select
Defer DataFrame::select: project the frame down to exactly the
evaluated expressions (an all-scalar selection collapses to one row,
like its eager counterpart).
LazyFrame::slice
Defer DataFrame::slice: a saturating length-first row window, an
omitted length running to the end. Like eager slice this never
fails — saturation resolves against the collected frame's height.
LazyFrame::sort
Defer DataFrame::sort: reorder rows by one or more
(key, descending, nulls_last) keys, later keys breaking ties. Each key
is an Expr evaluated over the whole frame at collect time, so a
missing column or a dtype clash only surfaces then.
LazyFrame::std
Defer DataFrame::std: a 1-row frame of each numeric column's sample
standard deviation (ddof = 1) as Float; a column with fewer than two
non-null cells — like a non-numeric one — takes a Null cell.
LazyFrame::sum
Defer DataFrame::sum: collapse the plan to a 1-row frame of each column's
sum (numeric columns their sum in the source dtype, non-numeric columns a
Null cell). collect equals the eager df.sum().
LazyFrame::tail
Defer DataFrame::tail: the last n rows, with the same total clamp
as head.
LazyFrame::top_k
Defer DataFrame::top_k: the k rows largest under by, reverse~
flipping the end. One node, not a sort+head composition, so the eager
verb's argument validation (k >= 0, a non-empty by) surfaces at
collect with the eager error.
LazyFrame::unique
Defer DataFrame::unique: drop duplicate rows, keeping survivors in their
original order. subset and keep mirror the eager verb — subset picks
the columns forming the duplicate key (every column when omitted, and the
output always carries all columns), First (the default) keeps each key's
first occurrence, Last its last, and None keeps only rows that occur
exactly once. Total like every builder: an unknown subset name surfaces
as ColumnNotFound at collect.
LazyFrame::unpivot
Defer DataFrame::unpivot: the wide → long reshape, the on? columns
folding into a variable_name? / value_name? pair under the index?
columns. Building stays total; the name-only column resolution and the
dtype unification surface as the eager verb's errors at collect.
LazyFrame::variance
Defer DataFrame::variance: the ddof = 1 variance, std's
Float-or-Null cell rules. Spelled variance because var is a
reserved MoonBit word.
LazyFrame::with_columns
Defer DataFrame::with_columns: derive new columns (or replace
same-named ones) from expressions, keeping every existing column.
LazyFrame::with_row_index
Defer DataFrame::with_row_index: prepend an Int counter column named
name (default "index") running from offset. Total like every builder
— a name that collides with an existing column surfaces as
DuplicateColumn at collect, and an offset the counter would overflow
as InvalidOperation there.
LazyGroupBy
pub struct LazyGroupBy {
// private fields
}A deferred group_by awaiting its aggregations — the lazy mirror of
the eager GroupedDataFrame stage in group_by(keys).agg(…),
except it holds no groups: just the input plan and the key
expressions, which agg completes into a single Aggregate node. Both
fields stay private — a LazyGroupBy is only ever the step between
LazyFrame::group_by and LazyGroupBy::agg, and sharing one across
several agg calls just forks the plan, like every other builder.
LazyGroupBy::agg
Complete the deferred group-by: defer group_by(keys).agg(exprs)
as one Aggregate node and return to the LazyFrame chain. Each
expression must be reduction-shaped (aggregations / literals and their
combinators — a bare column reference is not); building stays total,
so a non-reduction expression, a missing column, a dtype clash, or an
output-name collision waits for collect and surfaces as the eager
agg error (InvalidOperation / ColumnNotFound /
TypeMismatch / DuplicateColumn).
LazyGroupBy::count
Defer the eager group_by(keys).count(): each non-key column's non-null
cell count per group (the row count regardless of nulls is len).
LazyGroupBy::first
Defer the eager group_by(keys).first(): each non-key column's first
cell per group, in group row order.
LazyGroupBy::last
Defer the eager group_by(keys).last(): each non-key column's last cell
per group, in group row order.
LazyGroupBy::len
Defer the eager group_by(keys).len(): the single Int column "len"
holding each group's row count — nulls included, dtype-blind.
LazyGroupBy::max
Defer the eager group_by(keys).max(): the maximum counterpart of
min.
LazyGroupBy::mean
Defer the eager group_by(keys).mean(): each non-key column's mean as
Float per group.
LazyGroupBy::median
Defer the eager group_by(keys).median(): each non-key column's median
per group.
LazyGroupBy::min
Defer the eager group_by(keys).min(): each non-key column's minimum
per group, total over every dtype (NaN skipped).
LazyGroupBy::n_unique
Defer the eager group_by(keys).n_unique(): each non-key column's
distinct value count per group (a null counts as one distinct value,
every NaN one bucket).
LazyGroupBy::quantile
fn LazyGroupBy::quantile(self : LazyGroupBy, q : Double, interpolation? : QuantileInterpolation) -> LazyFrameDefer the eager group_by(keys).quantile(q, interpolation?): each
non-key column's q-th quantile under interpolation (default
Nearest) per group.
LazyGroupBy::std
Defer the eager group_by(keys).std(): each non-key column's sample
standard deviation (ddof = 1) per group.
LazyGroupBy::sum
Defer the eager group_by(keys).sum(): the sum of every non-key column
per group (numeric columns keep their dtype, an all-null group the
additive identity).
LazyGroupBy::variance
Defer the eager group_by(keys).variance(): the ddof = 1 variance,
std squared (var is a reserved MoonBit word).
scan_csv
The lazy CSV source: scan_csv(path) builds a LazyFrame whose leaf is a
deferred read of path — the push-down-aware counterpart of eager
read_csv. Nothing is read or parsed until collect: the builder is
total, like every other LazyFrame constructor, so a missing file or a
malformed CSV only surfaces then (as the eager reader's IoError /
ParseError / DuplicateColumn). It is not a streaming read — the reader
still tokenises the whole file, and streaming it is tracked as future
work.
The payoff over LazyFrame::LazyFrame(read_csv(path)) is that this leaf
absorbs both push-downs, so the work it skips is work the eager pipeline
does:
- projection — the optimizer narrows the leaf to the columns the pipeline provably consumes, and only those are parsed, so scan_csv("sales.csv").select([col("region"), col("revenue")]).collect() never builds the columns it drops;
- predicate — a filter sitting on the leaf moves into the read: the reader builds the predicate's columns, asks which rows survive, then parses the remaining columns for the survivors alone. Only the first such filter is absorbed; a second stays a node above the scan.
Both prune cells an eager read-then-filter would have parsed, which is the
one way a narrowed scan diverges from a full eager read: a ParseError
confined to a dropped column, or to a row the predicate drops in a column the
predicate does not read, does not surface. Dtype inference still walks the
whole file, so dtypes and the surviving cells match the eager read exactly.
options (delimiter, header, null strings, inference window, parse-error
policy, strict quotes, dtype pins, …) defaults to CsvReadOptions::CsvReadOptions(), mirroring eager
read_csv. It is captured into the plan and applied by the reader at
collect time; projection push-down narrows which columns those options
parse, never which options apply. The projection starts None (read every
column) — only the optimizer fills it.
scan_json
The lazy JSON source: scan_json(path) builds a LazyFrame whose leaf is
a deferred read of path — the push-down-aware counterpart of eager
read_json, for the single top-level array of records
([{...}, {...}]). Like every LazyFrame constructor it is total:
nothing is read or parsed until collect, so a missing file, malformed
JSON, or a non-object record only surfaces then (as the eager reader's
IoError / ParseError).
The one-way shape of the format sets what the push-downs can prune: the
top-level array must be parsed whole to find its records, so the pruning
both push-downs bring is at the assembly — projection leaves the
dropped columns uninferred and unbuilt, and the first predicate sitting
on the leaf builds only the surviving rows' cells past the predicate's
own columns. The parse itself is never skipped, which is also why this
is not a streaming read. A ParseError confined to a dropped column
or to a dropped row therefore does not surface, and the surviving cells
and dtypes match an eager read exactly — the contract the other two
scan sources honour, at the granularity this format allows.
options defaults to JsonReadOptions::JsonReadOptions(), mirroring eager
read_json, and is captured into the plan and applied at collect time;
projection push-down narrows which columns those options parse, never
which options apply.
scan_ndjson
The lazy NDJSON source: scan_ndjson(path) builds a LazyFrame whose leaf
is a deferred read of path — the push-down-aware counterpart of eager
read_ndjson, and the line-oriented sibling of scan_csv. Like every
LazyFrame constructor it is total: nothing is read or parsed until
collect, so a missing file or a malformed line only surfaces then (as the
eager reader's IoError / ParseError). It is not a streaming read —
the reader still parses every line — and it absorbs the same two push-downs
scan_csv documents: projection (only the consumed columns are built) and
the first predicate sitting on the leaf (the surviving rows' cells are the
only ones built past the predicate's own columns). So a parse error confined
to a dropped column, or to a dropped row, does not surface; the surviving
cells and the inferred dtypes match an eager read exactly. (The
single-array shape has its own source, scan_json.)
options (inference window, parse-error policy, dtype pins, …) defaults to
JsonReadOptions::JsonReadOptions(), mirroring eager read_ndjson. It is captured into the
plan and applied by the reader at collect time; projection push-down narrows
which columns those options parse, never which options apply. The projection
starts None (read every column) — only the optimizer fills it.
Install
Download zipA lightweight DataFrame and tabular-data library for MoonBit
dataframedata-analysiscsvtabular
Apache-2.0
Updated 6 days ago
160 downloads