#moonrockz/ion

Status: early. The core data model, the Ion text and Ion binary encodings, Ion Hash (Ion Hash 1.0 over SHA-256), Ion Schema 1.0 and 2.0 validation, and JSON interoperability are implemented and tested, with asynchronous streaming readers and writers for both encodings over moonbitlang/async.

#Installation

moon add moonrockz/ion

#Packages

#Repository layout

• pkgs/ is the module root package, moonrockz/ion, and it is the ion executable — so the published package is runnable at the short coordinate moonx moonrockz/ion;
• pkgs/ion/ is the @ion core data model (moonrockz/ion/ion);
• pkgs/text (with the async pkgs/text/stream), pkgs/hash, pkgs/binary (with the async pkgs/binary/stream), pkgs/json, and pkgs/schema are the remaining library packages.

#Quick start

let value = @text.read_ion("{ name: \"ion\", tags: [a, b] }")
println(@text.write_ion(value)) // {name: "ion", tags: [a, b]}

let values = @text.read_ion_datagram("{ name: \"ion\" } 42")
let bytes = @binary.write_binary(values) // Ion binary, with a local symbol table
let decoded = @binary.read_binary(bytes) // back to the same values

let value = @text.read_ion("{ name: \"ion\", tags: [a, b] }")
let json = @json.to_json(value) // {"name":"ion","tags":["a","b"]}
let ion = @json.to_ion(json)     // back to the Ion data model

let value = @ion.IonValue::from_fields([
  @ion.Field::new("name", @ion.IonValue::string("ion")),
  @ion.Field::new("tags", @ion.IonValue::list([
    @ion.IonValue::symbol("a"),
    @ion.IonValue::symbol("b"),
  ])),
])

let schema = @schema.Schema::load_from_text(
  "type::{ name: person, fields: { name: { type: string, occurs: required } } }",
)
let value = @text.read_ion("{ name: \"Ada\" }")
if schema.is_valid("person", value) {
  println("valid")
}

moon run pkgs -- print data.ion
moon run pkgs -- print --binary data.ion > data.10n
moon run pkgs -- hash data.ion
moon run pkgs -- validate schema.isl person data.ion

moonx moonrockz/ion version
moonx moonrockz/ion print data.ion
moonx moonrockz/ion hash data.ion
moonx moonrockz/ion validate schema.isl person data.ion

#Parsing APIs

let value = @text.read_ion("{a: 1}")               // DOM / AST
value.accept(visitor)                              // visitor
let total = value.fold(0, @ion.IonFold::default()) // fold
let tokens = @text.tokenize("int32::12")           // CST tokens
@text.parse_with_handler("1 2 3", handler)         // SAX (push)

#Streaming Ion text

let reader = @stream.TextReader::new(source) // moonrockz/ion/text/stream; any &@io.Reader
while reader.next() is Some(value) {
  println(@text.write_ion(value))
}
let writer = @stream.TextWriter::new(sink) // any &@io.Writer
writer.write(value)                        // one value per line
@stream.write_all(sink, values, pretty=true)

#Ion Hash

let value = @text.read_ion("{ name: \"ion\", tags: [a, b] }")
let digest = @hash.ion_hash_hex(value) // lowercase SHA-256
let bytes = @hash.ion_hash(value)      // 32 raw bytes

#JSON interoperability

let value = @text.read_ion("{ data: annot::{time: 1969-07-20T20:18Z}, n: 1.50 }")
let json = @json.to_json(value) // {"data":{"time":"1969-07-20T20:18Z"},"n":1.50}
let ion = @json.to_ion(json)     // back into the Ion data model

• JSON to Ion is faithful in the sense that a JSON value converted to Ion and back is the same JSON value, and it follows the cookbook's JSON-to-Ion rules: null, booleans, strings, arrays, and objects map to the matching Ion values, and a number becomes an Ion int, decimal, or float according to its spelling. The spelling core kept in repr wins when present — core keeps an integer literal above 2^53 - 1 and any literal that overflows a Double — so large integers survive exactly; otherwise the number's shortest round-trip spelling decides, so 1.50 becomes the Ion decimal 1.5 and 1e-7 an Ion float.
• Ion to JSON is lossy and follows the Ion cookbook's down-conversion process: a null of any type becomes null; integers and decimals keep their precision; nan and ±inf become null; timestamps and symbols become strings; a clob becomes a Latin-1 string and a blob a Base64 string; lists and s-expressions become arrays; a struct becomes an object; annotations are dropped. A symbol known only by symbol ID has no text, so converting it raises IonError::Unsupported rather than inventing one.

let value = @json.read_json("{\"a\": [1, 2]}") // {a: [1, 2]}
let text = @json.write_json(value)               // {"a":[1,2]}

#Design notes

• Annotations live on the value. Every IonValue carries an ordered annotation list plus a kind, so readers, writers, and validators reach annotations the same way.
• Absence is T?, never a sentinel. A SymbolToken has text : String? and sid : Int?; a decimal keeps an explicit negative-zero flag; a timestamp keeps an optional TimestampOffset.
• Exact numbers. Integers and decimals are exact (BigInt coefficient/exponent), so precision and signed zero round-trip through text.
• Decimal arithmetic is opt-in. add/subtract/multiply/compare/ to_double and conversion go through moonbitlang/x/decimal, whose decimal is normalized and caps the scale, so those helpers document what they drop; the value type itself keeps Ion's exact scale and signed zero.
• Derived precision. A timestamp's precision is derived from which components are present, not stored separately.
• One symbol table for both encodings. @ion.SymbolTable holds the symbols in effect in a stream. The text and binary readers apply a version marker ($ion_1_0 unquoted at the top level, or the binary marker) and a top-level $ion_symbol_table::{...} struct to it instead of returning them, and resolve each symbol ID against it. Imports of shared symbol tables resolve against an @ion.Catalog, which the readers and the CLI's --catalog option take. A symbol ID beyond the table is an error; one whose slot has no text keeps only its ID, and the writers keep that ID by reserving it in the symbol table they write.
• Unsupported is an error. The schema loader raises IonError::Unsupported for what it does not implement (imports when no resolver is given), rather than silently ignoring it.

#Roadmap

• Shared symbol tables: a symbol whose text a shared table leaves unknown keeps its symbol ID but not its place in that table, so any two such symbols compare as equal, as symbols with unknown text do.
• Ion 1.1 (#7): only Ion 1.0 is implemented.
• Ion binary output from the CLI (#8).
• An incremental writer that writes containers without building them first (#9).
• An event reader for Ion binary (#10).

#Building and testing

mise run setup       # fetch the test-data submodules and install dependencies
mise run test:check  # moon check
mise run test:unit   # moon test (unit, doc, snapshot, conformance, QuickCheck)
mise run test:all    # check + test
mise run build:native
moon fmt             # format
moon info            # regenerate package interfaces (*.mbti)
moon test --update   # refresh the golden fixtures' recorded output

#License

#The ion command-line tool (moonrockz/ion)

ion print [--binary | --pretty] [file|-]        Ion text or binary in; Ion text, pretty text, or binary out
ion json [file|-]                               Ion text or binary in, JSON out
ion fromjson [file|-]                           JSON in, Ion text out
ion hash [file|-]                               Ion Hash (SHA-256) of each value
ion validate <schema-file> <type-name> [file|-] validate each value
ion version                                     also `ion --version` or `ion -V`
ion help [command]                              also `ion --help` or `ion <command> --help`

#Examples

$ cat people.ion
{ name: "Ada", born: 1815 } { name: 42 }

$ ion print people.ion
{name: "Ada", born: 1815}
{name: 42}

$ ion print --binary people.ion > people.10n
$ ion print people.10n
{name: "Ada", born: 1815}
{name: 42}

$ ion json people.ion
{"name":"Ada","born":1815}
{"name":42}

$ echo '{"a": [1, 2.5]}' | ion fromjson
{a: [1, 25d-1]}

$ ion hash people.ion
5d2dbea37f03297ad83be8aa10f162c10e0c1e90fd950ce7b4972093774cdf2a
be93d3daaf4fe159a03cfe8e4b0c02394de41c4b2a690257f4d194e0ba1fc21e

$ cat person.isl
$ion_schema_2_0
type::{ name: person, type: struct, fields: { name: { type: string, occurs: required }, born: int } }

$ ion validate person.isl person people.ion
value 1: $.name: expected one of [string] but found int
1 of 2 value(s) failed validation
$ echo $?
1

#Behavior

• print, json, hash, and validate stream their Ion input through moonrockz/ion/text/stream and moonrockz/ion/binary/stream: each value is read, handled, and dropped in turn, so memory stays proportional to the largest value, not to the input.
• print --binary writes each value as it arrives, declaring each new symbol text once, in a local symbol table that appends to the ones before.
• An error stops a command after the values before it are handled, prints a message to stderr, and exits with status 1. So does a validation failure. Arguments that do not make a command print an error and the help of the command they concern to stderr, and exit with status 2.
• moonbitlang/core/argparse parses the arguments.
• fromjson reads its whole input, since a JSON document is one value.
• The tool builds for the native and wasm targets, which moonbitlang/async supports for files and stdin.