A jq implementation in MoonBit

json

query

Author

moonbit-community

Version

0.2.0

License

Apache-2.0

Repository

github.com/moonbit-community/moonbit-jq

Last updated

2 hours ago

Downloads

#MoonJQ — jq in MoonBit

MoonJQ is a high-performance, jq-compatible JSON query interpreter written in MoonBit. It implements a complete pipeline: lexer → parser → streaming interpreter with lazy evaluation using Iterator[Json].

#Why MoonJQ?

Streaming semantics - Process large JSON with constant memory via iterators
jq-compatible - Familiar syntax and behavior for jq users
Type-safe - Built with MoonBit's strong type system
Well-tested - 415+ passing tests covering core jq functionality
Documented - All code examples in this README are type-checked and tested

#Features

#Core Operations

Identity & Access: . (identity), .foo (field), .[0] (index), .[-1] (negative index)
Iteration: .[] (array iteration), .[2:4] (slicing), .. (recursive descent)
Composition: | (pipe), , (comma/multiple outputs)
Safety: ? (optional access), // (alternative/default)

#Operators

Arithmetic: + (add/concat), - (subtract), * (multiply/repeat), / (divide), % (modulo)
Comparison: ==, !=, <, <=, >, >=
Logical: and, or, not
Type coercion: Automatic for arithmetic operations

#Control Flow

Conditionals: if ... then ... else ... end
Error handling: try ... catch ...
Variables: $var (read-only bindings)

#Built-in Functions

Transformation: map(expr), select(expr), sort, reverse, flatten, flatten(n), unique
Aggregation: add, min, max, length
Inspection: type, keys, values
Math: floor, sqrt
Utility: empty, not

#Construction

Arrays: [expr], [] (empty)
Objects: {key: value}, {} (empty)

#Quick Start

#Installation

# Clone the repository
git clone https://github.com/moonbit-community/moonbit-jq.git
cd moonbit-jq

# Run tests to verify installation
moon test

#Basic Usage

Use the jq helper function to evaluate queries:

///|
/// Helper function: Evaluate a jq query and return newline-separated results.
/// This mimics the command-line jq tool's behavior.
fn jq(query : String, input : String) -> String raise {
  let expr = @parser.parse(query)
  let json = @json.parse(input)
  @runtime.eval_all(@runtime.compile(expr), json)
  .map(fn(v) { @debug.to_string(v) })
  .join("\n")
}

#Command Line

The moonbit-community/jq workspace module publishes a portable WebAssembly command that can be run directly from Mooncakes:

printf '%s' '{"name":"Moon"}' | moonx moonbit-community/jq -r '.name'
moonx moonbit-community/jq -n -c '{ok: true, values: [1, 2]}'
moonx moonbit-community/jq -c '.items[]' data.json

moonx uses linear-memory WebAssembly by default and forwards every argument after moonbit-community/jq to the command. The command follows jq's argument order: the filter comes first, followed by zero or more input files. When no file is provided, input is read from stdin.

moon run cmd/jq --target native -- -c '.foo' data.json
cat data.json | moon run cmd/jq --target native -- -r '.name'
moon run cmd/jq --target native -- -n -c '{ok: true}'
moon run cmd/jq --target native -- -f filter.jq data.json

Build the release binary when you want to run it directly:

moon build --target native --release cmd/jq
_build/native/release/build/moonbit-community/jq/jq.exe -c '.items[]' data.json

Supported CLI options:

-c, --compact-output: print compact JSON.
-r, --raw-output: print strings without JSON quotes.
-f, --from-file FILE: read the filter from FILE.
-n, --null-input: run the filter once with null input.
-l, --logs: treat input as JSONL/NDJSON and skip non-JSON lines.

See TUTORIAL.md for a CLI walkthrough adapted from the jq tutorial.

#Examples

All examples below are executable and type-checked by moon check README.mbt.md.

#1. Filter and Project

Extract specific fields from objects that meet criteria:

inspect( jq(query, input), content=( #|Object({"name": String("Alice"), "email": String("alice@example.com")}) ), ) }


**Explanation**: The `select(.age >= 18)` filters users 18 or older, then `{name: .name, email: .email}` constructs new objects with only those fields.

### 2. Optional Access with Defaults

Handle missing fields gracefully using `?` and `//`:

```mbt check
  inspect(
    jq(query, input),
    content=(
      #|String("(unknown)")
    ),
  )
}

Explanation: The ? operator prevents errors when .user.name doesn't exist, and // provides a default value.

#3. Transform and Aggregate

inspect(jq(query, input), content="Number(12)") }


**Explanation**: `map(. * 2)` doubles each number, then `add` sums them all: `(1*2 + 2*2 + 3*2) = 12`.

### 4. Filter Logs by Level

Extract specific log messages based on severity:

```mbt check
///|
test "readme: extract error messages" {

///|
  inspect(
    jq(query, input),
    content=(
      #|String("disk full")
      #|String("timeout")
    ),
  )
}

Explanation: Streaming semantics produce multiple outputs. Each error-level event produces one result.

#5. Array Slicing and Manipulation

Work with array subsets using slicing:

///|
test "readme: array slicing" {
  let query = ".items[1:3] | reverse"
  let input =
    #|{ "items": [10, 20, 30, 40, 50] }
  inspect(
    jq(query, input),
    content=(
      #|Array([Number(30), Number(20)])
    ),
  )
}

Explanation: [1:3] extracts elements at indices 1-2 (20, 30), then reverse flips the order.

#6. Recursive Descent

Find all values at any depth using ..:

///|
test "readme: recursive descent" {
  let query = ".. | select(type == \"number\")"
  let input =
    #|{
    #|  "a": 1,
    #|  "b": { "c": 2, "d": { "e": 3 } }
    #|}
  inspect(
    jq(query, input),
    content=(
      #|Number(1)
      #|Number(2)
      #|Number(3)
    ),
  )
}

Explanation: .. recursively visits all values in the structure, then select filters only numbers.

#Project Structure

moonbit-jq/
├── moon.work              # Workspace listing all member modules
├── moon.mod               # moonbit-community/moonjq library module
├── README.mbt.md          # This file (executable documentation)
├── TUTORIAL.md            # CLI tutorial
├── ast/                   # AST + streaming evaluator + integration tests
├── cmd/jq/                # moonbit-community/jq native + Wasm executable module
├── cmd/jqlog/             # moonbit-community/jqlog native + Wasm executable module
├── parser/                # Parser (includes lexer)
├── tests/cram/            # mooncram CLI tests and input fixtures

#Development

#Running Tests

# Run all tests (415+ tests)
moon test

# Run specific package tests
moon test -p parser
moon test -p ast

# Type-check without running tests
moon check

# Type-check this README
moon check README.mbt.md

# Update test snapshots
moon test --update

#CLI Cram Tests

The CLI tests use moonx moonbit-community/mooncram@0.1.0. Local commands declare their module in each mooncram fence and are built from the checkout. Run each suite from the corresponding module directory; mooncram resolves local modules from its startup directory. Auxiliary tools use pinned cli/*@0.2.1 packages from Mooncakes. Input files are checked in under tests/cram/fixtures/.

# From the repository root
(cd cmd/jq && moonx moonbit-community/mooncram@0.1.0 test ../../tests/cram/cli.md ../../TUTORIAL.md)
(cd cmd/jqlog && moonx moonbit-community/mooncram@0.1.0 test ../../tests/cram/jqlog.md)

The default backend is Wasm. Add --target native to the jq and jqlog test commands to test their native builds; CI runs both backends. Remote tools always run as Wasm. Use check instead of test to validate documents without executing commands, or update --dry-run to preview expectation changes. Mooncram parses commands directly: use moonx for every pipeline segment, (no-eol) for output without a newline, and [N] for a nonzero exit status. Unmarked cases assert stdout and the exit status; stderr can be checked with paired @STDOUT and @STDERR sections.

#Code Quality

# Format code
moon fmt

# Generate package interfaces
moon info

# Check for warnings
moon check --target all

#Implementation Highlights

Streaming: Uses MoonBit's Iterator for lazy evaluation and constant memory
Parser: Hand-written recursive-descent parser with precedence climbing
Error handling: Leverages MoonBit's checked error system with raise
Testing: 415+ tests using MoonBit's snapshot testing (inspect)

#Feature Status & Compatibility

See FEATURES.md for supported syntax and built-in functions, partial implementations, compatibility extensions, and remaining gaps.

Contributions welcome!

#License

See LICENSE.

#Acknowledgments

Inspired by jq by Stephen Dolan
Built with MoonBit

#Public API Reference

The @moonjq package exposes a simple, high-level API:

#Types

Type	Description
Query	A compiled jq query that can be evaluated multiple times

#Functions

Function	Signature	Description
parse	(String) -> Query raise	Compile a jq query string into a reusable Query
eval	(Query, Json) -> Iter[Json] raise	Evaluate a query, streaming results lazily
eval_all	(Query, Json) -> Array[Json] raise	Evaluate a query, collecting all results
run	(String, String) -> Array[Json] raise	Parse and evaluate in one step
run_json	(Query, Json) -> Array[Json] raise	Evaluate against already-parsed JSON

#Query Methods

Method	Signature	Description
Query::eval	(Query, Json) -> Iter[Json] raise	Stream evaluation results
Query::eval_all	(Query, Json) -> Array[Json] raise	Collect all results
Query::eval_logs	(Query, String) -> Iter[Json] raise	Process NDJSON logs

#License

See LICENSE.

EvalError

source

using @moonbit-community/moonjq/runtime { type EvalError }

Runtime failures are explicit events in the streaming API.

Query

source

pub struct Query {
  // private fields
} derive(@moonbitlang/core/debug.Debug
Debug)

Compiled jq query with private, immutable execution-plan storage.

Query::eval

source

fn Query::eval(self : Query, input : Json) -> Iter[Result[Json, @moonbit-community/moonjq/runtime.EvalError
EvalError]]

Methods on Query for direct control.

Query::eval_all

source

fn Query::eval_all(self : Query, input : Json) -> Array[Json] raise @moonbit-community/moonjq/runtime.EvalError
EvalError

Evaluate this compiled query against one JSON input and collect every value it produces.

This is the eager counterpart to Query::eval. It is convenient when the full result set is small enough to keep in memory, while Query::eval should be preferred for callers that want to stream results.

Raises if evaluation fails, for example when the query applies an operation to an incompatible JSON type.

Query::eval_logs

source

fn Query::eval_logs(self : Query, logs : String) -> Iter[Result[Json, @moonbit-community/moonjq/runtime.EvalError
EvalError]]

Apply the query to newline-delimited JSON logs, skipping invalid lines.

Query::to_repr

source

fn Query::to_repr(Query) -> @moonbitlang/core/debug.Repr
Repr

Preserve method-style access to the derived debug representation.

eval

source

fn eval(query : Query, input : Json) -> Iter[Result[Json, @moonbit-community/moonjq/runtime.EvalError
EvalError]]

Evaluate a compiled query against an input JSON value and stream results.

eval_all

source

fn eval_all(query : Query, input : Json) -> Array[Json] raise @moonbit-community/moonjq/runtime.EvalError
EvalError

Evaluate a compiled query and collect all results eagerly.

parse

source

fn parse(query : String) -> Query raise

Compile a jq query string into a reusable Query.

run

source

fn run(query : String, input : String) -> Array[Json] raise

Parse + evaluate a jq query against a JSON string input, collecting results.

run_json

source

fn run_json(query : Query, input : Json) -> Array[Json] raise @moonbit-community/moonjq/runtime.EvalError
EvalError

Evaluate a compiled query against an already parsed Json input.

Source Files

Site source Report issue Packages Build queue Skills Statistics

moonjq

#MoonJQ — jq in MoonBit

#Why MoonJQ?

#Features

#Core Operations

#Operators

#Control Flow

#Built-in Functions

#Construction

#Quick Start

#Installation

#Basic Usage

#Command Line

#Examples

#1. Filter and Project

#3. Transform and Aggregate

#5. Array Slicing and Manipulation

#6. Recursive Descent

#Project Structure

#Development

#Running Tests

#CLI Cram Tests

#Code Quality

#Implementation Highlights

#Feature Status & Compatibility

#License

#Acknowledgments

#Public API Reference

#Types

#Functions

#Query Methods

#License

#EvalError

#Query

#Query::eval

#Query::eval_all

#Query::eval_logs

#Query::to_repr

#eval

#eval_all

#parse

#run

#run_json

Source Files

EvalError

Query

Query::eval

Query::eval_all

Query::eval_logs

Query::to_repr

eval

eval_all

parse

run

run_json