#bobzhang/ajson

• Pull API: next_event() returns one event at a time; next_value(), next_member() and skip_value() work on whole values from any position.
• Memory proportional to nesting depth: an explicit container stack, no recursion, with a configurable depth limit.
• Input arrives in chunks from any async source; tokens may span chunk boundaries, including surrogate pairs and multi-byte UTF-8 sequences.
• Strict RFC 8259, with the same accept/reject decisions and values as @json.parse, which the tests check differentially. Input strings must be well-formed UTF-16, as MoonBit strings are; a lone surrogate code unit in the input is rejected.
• One document, or a stream of them (newline-delimited JSON, concatenated documents).
• Errors carry an offset, line and column.

#Example

///|
async test "read an array of objects in chunks" {
  let parts = ["[{\"name\": \"a\", \"n\": 1}, {\"na", "me\": \"b\", \"n\": 2}]"]
  let mut i = 0
  let parser = @ajson.Parser::new(
    @lexbuf.AsyncLexbuf::from_fn(() => {
      if i < parts.length() {
        i += 1
        Some(parts[i - 1])
      } else {
        None
      }
    }),
  )
  guard parser.next_event() is Some(StartArray) else { fail("not an array") }
  let names = []
  while parser.next_value() is Some(Object(members)) {
    if members.get("name") is Some(String(name)) {
      names.push(name)
    }
  }
  debug_inspect(names, content="[\"a\", \"b\"]")
  assert_true(parser.next_event() is None)
}

let parser = @ajson.Parser::from_reader(@stdio.stdin.text, multiple_values=true)
parser.each_value(value => println(value.stringify()))

moon build --target native --release
./_build/native/release/build/cmd/example/example.exe count < big.json

#API

Parser::new(lexbuf : @lexbuf.AsyncLexbuf, max_depth? : Int = 1024,
            max_string_length? : Int, multiple_values? : Bool = false) -> Parser
Parser::from_reader[R : @io.TextReader](reader : R, ...same options) -> Parser

async Parser::next_event(self) -> Event?         // None at the end of input
async Parser::next_value(self) -> Json?          // next complete value
async Parser::next_member(self) -> (String, Json)?
async Parser::skip_value(self) -> Bool
async Parser::each_value(self, f : async (Json) -> Unit) -> Unit
Parser::position(self) -> Position
Parser::depth(self) -> Int

enum Event { StartObject, EndObject, StartArray, EndArray, Key(String), Value(Json) }
struct Position { offset : Int, line : Int, column : Int }
suberror ParseError {
  UnexpectedChar(Position, Char), UnexpectedEof(Position),
  UnpairedSurrogate(Position), DepthLimitExceeded(Position, Int),
  TooLong(Position, Int), Interrupted
}

• next_value() at the top level returns the next document. Inside an array it returns the next element, and None when the array ends (the EndArray is consumed). Inside an object use next_member(), which returns None at the EndObject. skip_value() has the contract of next_value() but builds nothing.
• Values are the ones @json.parse gives: duplicate keys keep the last value, and numbers carry repr exactly when @json.parse would.
• Position.offset counts UTF-16 code units from the start of the stream; line is 1-based and column 0-based, as in @json.Position; column counts UTF-16 code units.
• After a syntax error every later call raises it again, and the events before the error are returned first. An error raised by the source (an I/O error) is kept the same way, but the events of the token group being scanned when it happened may be lost: the fast scanner's lookahead fails before the group is committed. After a cancellation, later calls raise Interrupted. Either way, do not reuse the lexbuf afterwards.

#Options

• max_depth: the deepest nesting accepted (default 1024, as @json.parse).
• max_string_length: rejects strings, property names and number literals longer than this many UTF-16 code units. It also switches to a scanner that reads at most 64 characters per lexscan call, so the parser's memory is bounded by the depth, about one input chunk, and the token being decoded, which is at most max_string_length. That scanner is slower (see below). Without the option, a single token is buffered whole. Either way, next_value() builds the whole value it returns.
• multiple_values: the input is a sequence of values separated by optional whitespace. A top-level number, true, false or null must be followed by whitespace or the end of input, so 1 2 is two values but 1[2] is an error. Empty input has no values.

#How it works

#Performance

moon run --target native --release cmd/benchmark

#Where the time goes: async compilation

moon run --target native --release cmd/asynctax

1. Box on suspension, not on entry. Keep locals that are live across a suspension point as plain locals on the path that does not suspend. At a suspension that actually happens, spill all of them into one heap frame (one allocation, not one per local) and reload them on resumption. For lexscan over AsyncLexbuf the only suspension is the cold refill, so this removes the per-scan allocations.
2. A synchronous fast path for async lexscan. Generate the Lexbuf code for the buffered input first. If the DFA decides before reaching the end of the buffer, which is almost always, commit and run the case body as usual; otherwise restart the scan from its start with the current async code. No state crosses from one to the other, so the hot path is exactly the sync one, at the cost of code size.
3. Direct return for async calls that complete synchronously. Let an async call return its result directly, or a "suspended" marker, as Kotlin's COROUTINE_SUSPENDED convention does, and allocate the callee's frame and the caller's continuation only when the callee really suspends, as C# does (with ValueTask, a synchronous completion allocates nothing). This helps every layer of async functions, not only lexscan: ajson makes two or three async calls per event.
4. Inline small async wrappers (functions that only forward to one async call) so that they do not add a frame and continuations of their own.

#Tests

moon test --target native
moon test --target js
moon test --target wasm