#moonlake

#What it is

• A query engine over files, not a database. Register CSV/Parquet files as tables, run SQL, get columnar result batches. There is no storage engine, no server and no FFI binding to a database: the parser, binder, planner, vectorized operators, hash join, aggregates, windows and spill store are all in this repository
• Two data sources, read and written in-house. CSV types are inferred (integers widen instead of wrapping, a ragged row is refused by row number). The Parquet reader and writer handle a stated subset of the format and refuse the rest with a pointed error rather than guessing — docs/sources.md
• Native and browser from one build. The same packages produce a single native binary for the command line and a wasm-gc module for the playground; it also builds for js. Only the filesystem differs: a small private POSIX stub (platform/io) gives native reads and writes their byte ranges and streaming, and wasm-gc/js read whole files because their host has no ranged read
• Larger-than-memory queries degrade honestly. Under --memory, a blocking ORDER BY spills to sorted runs and streams its result back; large joins and GROUP BYs partition their inputs and aggregate one partition at a time. What cannot be made to fit is refused with the reason, not OOM-killed
• Verified against DuckDB. All 22 TPC-H queries pass on their official texts within a 1e-9 relative tolerance, plus property tests, a pushdown fuzzer and the Parquet testing corpus — see docs/verification.md

#Quickstart

git clone https://github.com/superbigcup325/moonlake && cd moonlake
moon update
moon run cmd/main -- exec --csv harness/data/sf001/lineitem.csv \
  "SELECT l_returnflag, sum(l_extendedprice) FROM lineitem WHERE l_shipdate <= date '1998-09-02' GROUP BY l_returnflag ORDER BY l_returnflag"
# l_returnflag|sum_2
# A|532348211.6499983
# ...

#Install the binary

moon build cmd/main --target native --release
install -m755 _build/native/release/build/cmd/main/main.exe ~/.local/bin/moonlake
moonlake exec --csv lineitem.csv --file q6.sql     # SQL from a file: no shell quoting

#Use it as a library

import {
  "superbigcup325/moonlake",
  "superbigcup325/moonlake/catalog",
  "superbigcup325/moonlake/sources/csv",
}

// a @catalog.Catalog with registered table entries — @csv/@parquet
// scan into one; note the argument order: content/name, not name/content.
// registration is lazy: the schema is available at once, and a query decodes
// only the columns it projects, in 2048-row batches

///|
fn main raise {
  let cat = @catalog.Catalog::new()
  cat.register(@csv.scan_string("region,amount\nnorth,100\n", "events"))

  // read results back cell by cell: names() gives the columns,
  // row_count() the rows, get(row, col) a @types.Scalar (null via Null)
  let result = @moonlake.execute(
    "SELECT region, count(*) FROM events GROUP BY region", cat,
  )
  for r in 0..<result.row_count() {
    println(result.get(r, 0).to_string() + "|" + result.get(r, 1).to_string())
  }
}

#SQL surface

• SELECT (including *) / FROM (multi-table) / WHERE / GROUP BY / HAVING / ORDER BY / LIMIT, INNER / LEFT / RIGHT / FULL / CROSS joins, UNION [ALL] / INTERSECT / EXCEPT, derived tables, non-recursive WITH CTEs
• Expressions, aggregates (count(DISTINCT), median, string_agg, and the explicit approx_count_distinct / approx_quantile sketches), window functions, and correlated subqueries anywhere an expression may appear — three-valued NULL logic throughout, integer overflow raises instead of wrapping
• Semantics follow DuckDB and PostgreSQL where the engines could differ — rounding, division, integer-literal widening, float → integer casts — and every deviation from C/SQLite/MySQL is stated

#Verification

uv run --with duckdb python harness/gen_tpch.py   # TPC-H SF0.01 data + goldens
bash harness/check_all.sh                         # 33 goldens: all PASS (22 TPC-H + parquet + encoding variants)
MOONLAKE_DATA=harness/data/sf01 bash harness/check_all.sh   # same at SF0.1 (regenerate first: gen_tpch.py 0.1)

#Performance, for information

#Docs

• Playground (open, the plan panel and a result): register several CSV/Parquet files in a visible session, run cross-file joins, inspect the physical plan, drop tables or clear the session — the whole engine as wasm-gc, all client-side
• docs/DEMO.md: the reproducible demo, from running the engine to the DuckDB cross-check and the benchmark
• docs/cli.md: the moonlake command — inputs, output modes, plans, memory budgets and spilling
• docs/sql-surface.md: the SQL that is accepted, the semantics it follows, and what is refused
• docs/sources.md: CSV inference and ranged scans, the Parquet reader and writer scope, streaming IO per target
• docs/streaming.md: stream / next() — which shapes are pulled rather than collected, and the lifecycle
• docs/verification.md: the correctness and performance report (differential and property testing, pushdown fuzzing, memory and spill numbers)
• docs/bench-baseline.md: the release performance baseline and the machine it belongs to, with a committed JSON record and a --compare mode
• docs/spill-survey.md: how six systems handle larger-than-memory queries, and where moonlake's design lands
• docs/devlog.md: the build log, week by week
• docs/ROADMAP.md: what is being worked towards, what is refused on purpose, and what would start a direction currently on hold
• CHANGELOG.md: release notes

#Where moonlake sits

#Repository layout

#Development

moon check           # type-check
moon test            # run tests on the preferred target (wasm-gc)
moon test --target native  # filesystem/native backend tests
moon fmt             # format
moon run cmd/main    # run the CLI
git config core.hooksPath hooks   # once per clone: pre-commit gate

moon update   # refresh the registry index; the graph downloads on the first check
moon check

ln -s ../colenc .colenc
printf 'members = [".", "./.colenc"]\n' > moon.work

#Acknowledgements

• moonbitlang/x (Apache-2.0) — filesystem, argument parsing and encoding utilities
• moonbitlang/quickcheck (Apache-2.0) — property-based testing
• superbigcup325/colenc (Apache-2.0) — Parquet DELTA, BYTE_STREAM_SPLIT and RLE encoding/decoding primitives
• apache/parquet-format (Apache-2.0) — the Parquet format and encoding specifications that sources/parquet implements; its parquet-testing corpus supplies the files the reader is verified against
• The TPC-H query texts in harness/queries/ are the official benchmark texts, used here for correctness validation against DuckDB; TPC-H is a trademark of the Transaction Processing Performance Council

#License