Embeddable analytical query engine for MoonBit: run SQL over CSV/Parquet files, native and WebAssembly from one codebase.

sql

olap

query-engine

csv

parquet

columnar

Download zip

Author

superbigcup325

Version

0.1.0

License

Apache-2.0

Repository

github.com/superbigcup325/moonlake

Last updated

2 hours ago

Downloads

Dependencies

#moonlake

Embeddable analytical query engine for MoonBit — run SQL over CSV/Parquet files in-process, with native and WebAssembly builds from one codebase.

Status: v1 SQL surface complete, under acceptance hardening. Projection pruning and the vectorized evaluator are the remaining v1 items; APIs may still change.

#What it is

moonlake is a pure-MoonBit, columnar SQL query engine for analytical (OLAP) workloads:

Register CSV/Parquet files as tables, run SQL, get columnar result batches.
Ships as a native single binary for ad-hoc command-line analysis, embeds as a MoonBit library, and compiles to WebAssembly (GC) from the same codebase.
No FFI, no external database process, no storage engine — moonlake reads external files and computes in memory.

#v1 SQL surface

SELECT / FROM (multi-table) / WHERE / GROUP BY / HAVING / ORDER BY / LIMIT
INNER / LEFT / CROSS joins (hash join; join order chosen greedily by connectivity, LEFT keeps written order)
Expressions: arithmetic, comparison, CASE WHEN, IN, NOT IN, BETWEEN, LIKE, NOT LIKE, EXTRACT(year/month/day), DATE literals, three-valued NULL logic throughout
Aggregates: sum / avg / min / max / count / count(*) / count(DISTINCT)
Derived tables (FROM (SELECT ...) AS t), non-correlated scalar subqueries and IN / NOT IN (SELECT ...)
Pushdown: single-table predicates from WHERE/ON are applied as build/probe filters at each hash join; equalities become hash keys even when implied by disjunctions

Out of scope for v1: writes (INSERT/UPDATE/DDL), persistence, transactions, indexes, correlated subqueries, window functions, cost-based optimization.

#TPC-H cross-validation

SQL correctness is validated by differential testing against DuckDB over the TPC-H benchmark: 14 of the 22 queries run — 11 on their official text, 3 with the sanctioned rewrites (derived tables instead of the CTE, a scalar subquery for the max) — every result matching DuckDB within 1e-9 relative tolerance on SF0.01:

direct	rewritten
Q1 Q3 Q5 Q6 Q7 Q8 Q10 Q11 Q12 Q13 Q14 Q16 Q19	Q15 (CTE inlined as a derived table)

The whole chain is reproducible and runs in CI (job tpch regenerates the data and DuckDB's answers from scratch, then diffs every query):

uv run --with duckdb python harness/gen_tpch.py   # TPC-H SF0.01 data + goldens
bash harness/check_all.sh                         # 15 goldens: all PASS

Informational benchmark (SF0.01, best of 3, via harness/bench.py; row-at-a-time evaluation — vectorization is the declared next step):

query	moonlake native	duckdb
Q6	539 ms	0.7 ms
Q1	622 ms	3.0 ms
Q3	713 ms	4.8 ms
Q7	995 ms	4.3 ms

#Quickstart

git clone https://github.com/superbigcup325/moonlake && cd moonlake
moon run cmd/main -- exec --csv harness/data/sf001/lineitem.csv \
  "SELECT l_returnflag, sum(l_extendedprice) FROM lineitem WHERE l_shipdate <= date '1998-09-02' GROUP BY l_returnflag ORDER BY l_returnflag"
# l_returnflag|sum_2
# A|532348211.6499983
# ...

Joins take one --csv per table; --parquet scans Parquet files (declare epoch-day DATE columns with --date-col); --explain prints the physical plan; --json emits machine-readable output.

As a library:

moon add superbigcup325/moonlake

// register tables, run SQL, consume the columnar result
let result = @moonlake.execute(
  "SELECT region, count(*) FROM events GROUP BY region",
  cat, // a @catalog.Catalog with registered table entries
)

#Where moonlake sits

MoonBit's ecosystem had SQL parsers and format readers, but no query engine over them. moonlake fills the compute layer and is designed to sit next to, not on top of, its neighbours:

project	what it is	boundary with moonlake
moonbit-community/sqlparser	SQL lexer/parser (AST)	parsing only, no execution; moonlake ships an in-house subset front-end today (sqlparser's select-statement AST is not destructurable cross-package yet) and stays pinned as the future swap-in once its visibility improves
moonbit-community/NyaCSV	CSV dialect parser	text parsing only; moonlake's CSV source builds typed columnar batches on top of it
mizchi/parquet	Parquet reader/writer	format decoding only; moonlake adapts its columnar read into the same vectors the executor consumes
shunge/arrow (MoonArrow)	Arrow IPC format read/write	memory-format interchange; a future to_arrow bridge is cooperation, not competition
uiwcvb/moonsql	embedded OLTP database (row storage, CRUD, persistence)	different species — SQLite to moonlake's DuckDB: transactional storage vs external-file analytics
f4ah6o/duckdb, mizchi/duckdb	DuckDB C++ bindings	FFI route: the wasm-gc target is a stub upstream and the binding build is unstable; moonlake is pure MoonBit and runs natively in the browser

#Development

moon check           # type-check
moon test            # run tests
moon fmt             # format
moon run cmd/main    # run the CLI
git config core.hooksPath hooks   # once per clone: pre-commit gate

The pre-commit hook re-runs interface freshness (moon info), formatting, moon check --deny-warn and the tests before every commit — the same gates CI runs. CI covers native / wasm-gc / js test matrices plus the TPC-H cross-validation job.

#Acknowledgements

moonlake is being built on top of these open-source MoonBit packages:

moonbit-community/sqlparser (Apache-2.0) — SQL lexer/parser
moonbit-community/NyaCSV (Apache-2.0) — CSV parsing
mizchi/parquet (Apache-2.0) — Parquet reading
moonbitlang/quickcheck (Apache-2.0) — property-based testing

#License

Apache-2.0

DESCRIPTION

source

let DESCRIPTION : String

Short human-readable description of the module.

VERSION

source

let VERSION : String

Version of the superbigcup325/moonlake module, kept in sync with moon.mod.

execute

source

fn execute(sql : String, catalog : @superbigcup325/moonlake/catalog.Catalog
Catalog) -> @superbigcup325/moonlake/engine.SelectResult
SelectResult raise @superbigcup325/moonlake/types.SqlError
SqlError