moonjq

A jq implementation in MoonBit

jq
json
query
moon add bobzhang/moonjq@0.1.1
Download zip
Author
Version
0.1.1
License
Apache-2.0
Last updated
3 hours ago
Downloads
145

Dependencies

README

#MoonJQ — jq in MoonBit

Test Status MoonBit

MoonJQ is a high-performance, jq-compatible JSON query interpreter written in MoonBit. It implements a complete pipeline: lexer → parser → streaming interpreter with lazy evaluation using Iterator[Json].

#Why MoonJQ?

  • Streaming semantics - Process large JSON with constant memory via iterators
  • jq-compatible - Familiar syntax and behavior for jq users
  • Type-safe - Built with MoonBit's strong type system
  • Well-tested - 415+ passing tests covering core jq functionality
  • Documented - All code examples in this README are type-checked and tested

#Features

#Core Operations

  • Identity & Access: . (identity), .foo (field), .[0] (index), .[-1] (negative index)
  • Iteration: .[] (array iteration), .[2:4] (slicing), .. (recursive descent)
  • Composition: | (pipe), , (comma/multiple outputs)
  • Safety: ? (optional access), // (alternative/default)

#Operators

  • Arithmetic: + (add/concat), - (subtract), * (multiply/repeat), / (divide), % (modulo)
  • Comparison: ==, !=, <, <=, >, >=
  • Logical: and, or, not
  • Type coercion: Automatic for arithmetic operations

#Control Flow

  • Conditionals: if ... then ... else ... end
  • Error handling: try ... catch ...
  • Variables: $var (read-only bindings)

#Built-in Functions

  • Transformation: map(expr), select(expr), sort, reverse, flatten, flatten(n), unique
  • Aggregation: add, min, max, length
  • Inspection: type, keys, values
  • Math: floor, sqrt
  • Utility: empty, not

#Construction

  • Arrays: [expr], [] (empty)
  • Objects: {key: value}, {} (empty)

#Quick Start

#Installation

# Clone the repository git clone https://github.com/moonbit-community/moobit-jq.git cd moobit-jq # Run tests to verify installation moon test

#Basic Usage

Use the jq helper function to evaluate queries:

///|
/// Helper function: Evaluate a jq query and return newline-separated results.
/// This mimics the command-line jq tool's behavior.
fn jq(query : String, input : String) -> String raise {
let expr = @parser.parse(query)
let json = @json.parse(input)
@ast.eval(expr, json).collect().map(fn(v) { @debug.to_string(v) }).join("\n")
}

#Command Line

The native command follows jq's argument order: the filter comes first, followed by zero or more input files. When no file is provided, input is read from stdin.

moon run cmd/jq --target native -- -c '.foo' data.json cat data.json | moon run cmd/jq --target native -- -r '.name' moon run cmd/jq --target native -- -n -c '{ok: true}' moon run cmd/jq --target native -- -f filter.jq data.json

Build the release binary when you want to run it directly:

moon build --target native --release cmd/jq _build/native/release/build/cmd/jq/jq.exe -c '.items[]' data.json

Supported CLI options:

  • -c, --compact-output: print compact JSON.
  • -r, --raw-output: print strings without JSON quotes.
  • -f, --from-file FILE: read the filter from FILE.
  • -n, --null-input: run the filter once with null input.
  • -l, --logs: treat input as JSONL/NDJSON and skip non-JSON lines.

See TUTORIAL.md for a CLI walkthrough adapted from the jq tutorial.

#Examples

All examples below are executable and type-checked by moon check README.mbt.md.

#1. Filter and Project

Extract specific fields from objects that meet criteria:

inspect( jq(query, input), content=( #|Object({"name": String("Alice"), "email": String("alice@example.com")}) ), ) }
**Explanation**: The `select(.age >= 18)` filters users 18 or older, then `{name: .name, email: .email}` constructs new objects with only those fields. ### 2. Optional Access with Defaults Handle missing fields gracefully using `?` and `//`: ```mbt check inspect( jq(query, input), content=( #|String("(unknown)") ), ) }

Explanation: The ? operator prevents errors when .user.name doesn't exist, and // provides a default value.

#3. Transform and Aggregate

inspect(jq(query, input), content="Number(12)") }
**Explanation**: `map(. * 2)` doubles each number, then `add` sums them all: `(1*2 + 2*2 + 3*2) = 12`. ### 4. Filter Logs by Level Extract specific log messages based on severity: ```mbt check ///| test "readme: extract error messages" { ///| inspect( jq(query, input), content=( #|String("disk full") #|String("timeout") ), ) }

Explanation: Streaming semantics produce multiple outputs. Each error-level event produces one result.

#5. Array Slicing and Manipulation

Work with array subsets using slicing:

///|
test "readme: array slicing" {
let query = ".items[1:3] | reverse"
let input =
#|{ "items": [10, 20, 30, 40, 50] }
inspect(
jq(query, input),
content=(
#|Array([Number(30), Number(20)])
),
)
}

Explanation: [1:3] extracts elements at indices 1-2 (20, 30), then reverse flips the order.

#6. Recursive Descent

Find all values at any depth using ..:

///|
test "readme: recursive descent" {
let query = ".. | select(type == \"number\")"
let input =
#|{
#| "a": 1,
#| "b": { "c": 2, "d": { "e": 3 } }
#|}
inspect(
jq(query, input),
content=(
#|Number(1)
#|Number(2)
#|Number(3)
),
)
}

Explanation: .. recursively visits all values in the structure, then select filters only numbers.

#Project Structure

moobit-jq/ ├── moon.mod # Module metadata ├── README.mbt.md # This file (executable documentation) ├── TUTORIAL.md # CLI tutorial ├── ast/ # AST + streaming evaluator + integration tests ├── cmd/jq/ # Native jq-compatible CLI ├── parser/ # Parser (includes lexer) ├── tests/cram/ # Moon Cram CLI tests

#Development

#Running Tests

# Run all tests (415+ tests) moon test # Run specific package tests moon test -p parser moon test -p ast # Type-check without running tests moon check # Type-check this README moon check README.mbt.md # Update test snapshots moon test --update

#CLI Cram Tests

The CLI tests use the moon cram command. moon cram builds the workspace first and puts the built CLI binaries in PATH, so the cram examples call jq.exe directly.

moon cram test tests/cram TUTORIAL.md

#Code Quality

# Format code moon fmt # Generate package interfaces moon info # Check for warnings moon check --target all

#Implementation Highlights

  • Streaming: Uses MoonBit's Iterator for lazy evaluation and constant memory
  • Parser: Hand-written recursive-descent parser with precedence climbing
  • Error handling: Leverages MoonBit's checked error system with raise
  • Testing: 415+ tests using MoonBit's snapshot testing (inspect)

#Limitations & Roadmap

See FEATURES.md for detailed feature status.

Not yet implemented:
  • Variable binding with as patterns
  • reduce expressions
  • sort_by, group_by
  • Assignment operators (|=, =)
  • String interpolation (\(expr))
  • Many string/array utility functions

Contributions welcome!

#License

See LICENSE.

#Acknowledgments

  • Inspired by jq by Stephen Dolan
  • Built with MoonBit

#Public API Reference

The @moonjq package exposes a simple, high-level API:

#Types

TypeDescription
QueryA compiled jq query that can be evaluated multiple times

#Functions

FunctionSignatureDescription
parse(String) -> Query raiseCompile a jq query string into a reusable Query
eval(Query, Json) -> Iter[Json] raiseEvaluate a query, streaming results lazily
eval_all(Query, Json) -> Array[Json] raiseEvaluate a query, collecting all results
run(String, String) -> Array[Json] raiseParse and evaluate in one step
run_json(Query, Json) -> Array[Json] raiseEvaluate against already-parsed JSON

#Query Methods

MethodSignatureDescription
Query::eval(Query, Json) -> Iter[Json] raiseStream evaluation results
Query::eval_all(Query, Json) -> Array[Json] raiseCollect all results
Query::eval_logs(Query, String) -> Iter[Json] raiseProcess NDJSON logs

#License

See LICENSE.

#
Query

Compiled jq query wrapper so the AST stays internal.

#
Query::eval

fn Query::eval(self : Query, input : Json) -> Iter[Json] raise

Methods on Query for direct control.

#
Query::eval_all

fn Query::eval_all(self : Query, input : Json) -> Array[Json] raise

Evaluate this compiled query against one JSON input and collect every value it produces.

This is the eager counterpart to Query::eval. It is convenient when the full result set is small enough to keep in memory, while Query::eval should be preferred for callers that want to stream results.

Raises if evaluation fails, for example when the query applies an operation to an incompatible JSON type.

#
Query::eval_logs

fn Query::eval_logs(self : Query, logs : String) -> Iter[Json] raise

Apply the query to newline-delimited JSON logs, skipping invalid lines.

#
Query::to_repr

Preserve method-style access to the derived debug representation.

#
eval

fn eval(query : Query, input : Json) -> Iter[Json] raise

Evaluate a compiled query against an input JSON value and stream results.

#
eval_all

fn eval_all(query : Query, input : Json) -> Array[Json] raise

Evaluate a compiled query and collect all results eagerly.

#
parse

fn parse(query : String) -> Query raise

Compile a jq query string into a reusable Query.

#
run

fn run(query : String, input : String) -> Array[Json] raise

Parse + evaluate a jq query against a JSON string input, collecting results.

#
run_json

fn run_json(query : Query, input : Json) -> Array[Json] raise

Evaluate a compiled query against an already parsed Json input.