#Asynchronous I/O (@moonbitlang/core/async/io)

    This package defines asynchronous byte and text streams for MoonBit. Binary I/O uses BinaryReader and BinaryWriter; text I/O uses TextReader and TextWriter. Keeping the two forms separate makes encoding explicit and keeps binary protocols free of implicit UTF-8 conversions.

    The package also provides in-memory pipes, a callback-backed MemoryReader, UTF-8 adapters, and buffered writing.

    #Quick start

    pipe() returns connected binary endpoints. Writes apply backpressure until a reader accepts the data, and closing the write end makes the reader reach EOF.

    ///|
    async test "binary pipe quick start" {
    @async.with_task_group() <| root => {
    let (reader, writer) = @io.pipe()
    defer reader.close()
    root.spawn_bg() <| () => {
    defer writer.close()
    writer.write(b"Hello, ")
    writer.write(b"MoonBit!")
    }
    inspect(@utf8.decode(reader.read_all()), content="Hello, MoonBit!")
    }
    }

    #Binary streams

    #BinaryReader

    BinaryReader owns a read-ahead buffer. Its methods provide three useful styles of reading:

    • read_into(dst, offset?, max_len?) reads into caller-owned storage and returns the number of bytes read. A zero return value indicates EOF.
    • read() returns an owned chunk as soon as data is available, or None at EOF.
    • read_at_least(min_len) returns an owned chunk containing at least min_len bytes, and may return more.
    • peek() returns a zero-copy view as soon as data is available, or None at EOF, without consuming it.
    • peek_at_least(min_len) returns a zero-copy view containing at least min_len bytes without consuming it.

    If EOF arrives before min_len bytes are buffered, read_at_least and peek_at_least raise ReaderClosed. Any shorter trailing data remains buffered and can still be obtained by a later operation with a smaller minimum.

    ///|
    async test "read into caller-owned storage" {
    @async.with_task_group() <| root => {
    let (reader, writer) = @io.pipe()
    defer reader.close()
    root.spawn_bg() <| () => {
    defer writer.close()
    writer.write(b"hello")
    }
    let dst = FixedArray::make(9, b'.')
    let count = reader.read_into(dst, offset=2, max_len=5)
    inspect(count, content="5")
    inspect(
    @utf8.decode(dst.unsafe_reinterpret_as_bytes()[2:7]),
    content="hello",
    )
    }
    }

    ///|
    async test "peek then consume an exact frame" {
    @async.with_task_group() <| root => {
    let (reader, writer) = @io.pipe()
    defer reader.close()
    root.spawn_bg() <| () => {
    defer writer.close()
    writer.write(b"HEADbody")
    }
    let preview = reader.peek_exactly(4).to_owned()
    inspect(@utf8.decode(preview), content="HEAD")
    inspect(@utf8.decode(reader.read_exactly(4)), content="HEAD")
    inspect(@utf8.decode(reader.read_all()), content="body")
    }
    }

    The view returned by peek, peek_at_least, peek_exactly, or peek_until is valid only until the next operation on that reader. Call .to_owned() if the bytes must outlive that operation.

    Other binary reader operations are:

    • read_exactly(len) and peek_exactly(len) wait for exactly len bytes and raise ReaderClosed if EOF arrives first.
    • drop_exactly(len) consumes exactly len bytes without returning them.
    • read_all() drains the stream into a Bytes value.
    • drop_all() drains the stream and returns the number of discarded bytes.
    • read_until(pattern, include_delimiter?) reads through the next delimiter.
    • peek_until(pattern, include_delimiter?) finds the same segment without consuming it.

    read_until always consumes a delimiter it finds. By default the delimiter is excluded from the result; set include_delimiter=true to retain it. EOF acts as an implicit delimiter, so a final unterminated segment is returned once, then the next call returns None.

    ///|
    async test "read delimiter-separated binary records" {
    @async.with_task_group() <| root => {
    let (reader, writer) = @io.pipe()
    defer reader.close()
    root.spawn_bg() <| () => {
    defer writer.close()
    writer.write(b"alpha|beta")
    }
    guard reader.read_until(b"|") is Some(first) else {
    fail("missing first record")
    }
    inspect(@utf8.decode(first), content="alpha")
    guard reader.read_until(b"|") is Some(second) else {
    fail("missing final record")
    }
    inspect(@utf8.decode(second), content="beta")
    assert_eq(reader.read_until(b"|"), None)
    }
    }

    #BinaryWriter

    write(data) accepts a BytesView and does not return until the complete view has been committed to the underlying writer. A slice can therefore be forwarded without first copying it into a new Bytes value.

    write_reader(reader) copies a binary reader into a writer until EOF.

    ///|
    async test "write a byte view" {
    @async.with_task_group() <| root => {
    let (reader, writer) = @io.pipe()
    defer reader.close()
    root.spawn_bg() <| () => {
    defer writer.close()
    let packet = b"xxpayloadyy"
    writer.write(packet[2:9])
    }
    inspect(@utf8.decode(reader.read_all()), content="payload")
    }
    }

    write guarantees delivery to the underlying writer object, but not a destination-specific durability property such as an operating-system file flush.

    #Text streams and UTF-8 adapters

    Text is opt-in. Utf8Reader adapts a BinaryReader to TextReader, and Utf8Writer adapts a BinaryWriter to TextWriter.

    • TextReader::read_all() decodes all remaining bytes as UTF-8.
    • TextReader::read_until(pattern, include_delimiter?) searches for a text delimiter and returns decoded text.
    • TextWriter::write(text) UTF-8 encodes a StringView.
    • TextWriter::write_json(json) writes the JSON text representation.

    ///|
    async test "UTF-8 text adapters" {
    @async.with_task_group() <| root => {
    let (reader, writer) = @io.pipe()
    defer reader.close()
    root.spawn_bg() <| () => {
    defer writer.close()
    let text_writer = @io.Utf8Writer(writer)
    text_writer.write("你好|MoonBit")
    }
    let text_reader = @io.Utf8Reader(reader)
    guard text_reader.read_until("|") is Some(first) else {
    fail("missing first text record")
    }
    inspect(first, content="你好")
    inspect(text_reader.read_all(), content="MoonBit")
    }
    }

    Decoding invalid UTF-8 raises an error. Use the binary interfaces when arbitrary bytes must be preserved.

    #MemoryReader

    MemoryReader turns an asynchronous writer callback into a binary reader. The callback runs in a background coroutine; returning closes the stream, raising an error propagates it to the reader, and closing the reader cancels the callback.

    ///|
    async test "callback-backed memory reader" {
    let reader = @io.MemoryReader() <| writer => {
    writer.write(b"generated ")
    @async.sleep(10)
    writer.write(b"content")
    }
    defer reader.close()
    inspect(@utf8.decode(reader.read_all()), content="generated content")
    }

    #Buffered writing

    BufferedWriter::new(writer, size?) batches small writes in a fixed-size buffer. Call flush() before accessing the underlying destination through another path or before closing it. The buffer size must be positive.

    ///|
    async test "buffered writer" {
    @async.with_task_group() <| root => {
    let (reader, writer) = @io.pipe()
    defer reader.close()
    root.spawn_bg() <| () => {
    defer writer.close()
    let buffered = @io.BufferedWriter::new(writer, size=4)
    buffered.write(b"ab")
    buffered.write(b"cd")
    buffered.write(b"ef")
    buffered.flush()
    }
    inspect(@utf8.decode(reader.read_all()), content="abcdef")
    }
    }

    #Implementing an endpoint

    Application code normally depends on BinaryReader and BinaryWriter. Endpoint implementations use the lower-level traits:

    ///|
    pub(open) trait BinaryReaderImpl {
    fn get_controller(Self) -> BinaryReaderController
    async fn raw_read_impl(
    Self,
    FixedArray[Byte],
    offset~ : Int32,
    max_len~ : Int32,
    ) -> Int32
    }

    ///|
    pub(open) trait BinaryWriterImpl {
    fn get_controller(Self) -> BinaryWriterController
    async fn raw_write_impl(Self, Bytes, offset~ : Int32, len~ : Int32) -> Int32
    }

    A binary reader stores one BinaryReaderController, returns it from get_controller, implements raw_read_impl, and explicitly implements BinaryReader to opt into the default high-level methods. A writer similarly stores one BinaryWriterController, returns it from get_controller, implements raw_write_impl, and explicitly implements BinaryWriter. Writer controller internals are intentionally hidden so shared behavior can evolve without changing endpoint implementations or the high-level writer API.

    Raw operations communicate directly with the underlying endpoint. On success they must make positive progress, may return fewer bytes than requested, and must not call the high-level methods recursively. For readers, zero means EOF.

    #Usage notes

    • Run only one operation at a time on a given reader or writer. The interfaces do not serialize concurrent calls.
    • Use peek for parser lookahead, but copy a returned view before another reader operation if it must be retained.
    • Prefer read_into for caller-managed buffers, read for streaming chunks, and read_all only when the remaining input comfortably fits in memory.
    • Use Utf8Reader and Utf8Writer at the text boundary instead of mixing text conversion into binary protocol code.
    • Flush a BufferedWriter at protocol boundaries and before closing its underlying writer.

    BinaryReader

    pub(open) trait BinaryReader : BinaryReaderImpl {
    async fn read_into(Self, FixedArray[Byte], offset? : Int, max_len? : Int) -> Int = _
    async fn read(Self) -> Bytes? = _
    async fn read_at_least(Self, Int) -> Bytes = _
    async fn peek(Self) -> BytesView? = _
    async fn peek_at_least(Self, Int) -> BytesView = _
    async fn read_exactly(Self, len : Int) -> Bytes = _
    async fn drop_exactly(Self, len : Int) -> Unit = _
    async fn peek_exactly(Self, len : Int) -> BytesView = _
    async fn read_all(Self) -> Bytes = _
    async fn drop_all(Self) -> Int64 = _
    async fn read_until(Self, pattern : BytesView, include_delimiter? : Bool) -> Bytes? = _
    async fn peek_until(Self, pattern : BytesView, include_delimiter? : Bool) -> BytesView? = _
    }

    Trait for an asynchronous binary input stream. This trait is intended for direct use by end users. All methods in this trait have corresponding default implementations. Custom readers should implement BinaryReaderImpl and explicitly implement BinaryReader to opt in to these defaults. Methods in BinaryReader should not be overridden in general. When overridden for a reader-specific optimization, the alternative implementation should respect behavior of the default implementation.

    BinaryReaderImpl

    pub(open) trait BinaryReaderImpl {
    fn get_controller(Self) -> BinaryReaderController
    async fn raw_read_impl(Self, buf : FixedArray[Byte], offset~ : Int, max_len~ : Int) -> Int
    async fn ensure_buffered(Self, Int) -> Bool = _
    }

    BinaryReaderImpl contains the low-level operations used by the default implementations of BinaryReader methods. Methods in BinaryReaderImpl should not be invoked by users directly in general, and implementers should not expose BinaryReaderImpl methods via extend. In fact, it is recommended to keep impl BinaryReaderImpl private.

    Every reader implementation should store a BinaryReaderController for buffering, and get_controller should return that controller.

    raw_read_impl is the actual read implementation, it should ignore the controller and communicate with the underlying reader object directly, reading up to max_len bytes into buf[offset:]. raw_read_impl should return the number of bytes actually read into buf. raw_read_impl may read less than max_len bytes, but it should always read at least one byte on success, and a zero return value should always indicate EOF. raw_read_impl should return as soon as possible after at least one byte of data is read. It must not raise ReaderClosed to indicate its own EOF; return zero instead. A transformed reader should translate ReaderClosed raised by an inner reader into an error that describes the malformed or truncated transformed stream instead of propagating it.

    BinaryWriter

    pub(open) trait BinaryWriter : BinaryWriterImpl {
    async fn write(Self, BytesView) -> Unit = _
    async fn write_reader(Self, &BinaryReader) -> Unit = _
    }

    Trait for an asynchronous binary output stream. This trait is intended for direct use by end users. All methods in this trait have corresponding default implementations. Custom writers should implement BinaryWriterImpl and explicitly implement BinaryWriter to opt in to these defaults. Methods in BinaryWriter should not be overridden in general. When overridden for a writer-specific optimization, the alternative implementation should respect behavior of the default implementation.

    BinaryWriterImpl

    pub(open) trait BinaryWriterImpl {
    fn get_controller(Self) -> BinaryWriterController
    async fn raw_write_impl(Self, Bytes, offset~ : Int, len~ : Int) -> Int
    }

    BinaryWriterImpl contains the low-level operation used by the default implementations of BinaryWriter methods. Methods in BinaryWriterImpl should not be invoked by users directly in general, and implementers should not expose BinaryWriterImpl methods via extend. In fact, it is recommended to keep impl BinaryWriterImpl private.

    Every writer implementation should store a BinaryWriterController and return it from get_controller. The controller is reserved for behavior shared by the default implementations, such as locking and buffering.

    raw_write_impl is the actual write implementation, it should communicate with the underlying writer object directly, writing up to len bytes from buf[offset:] into the writer. raw_write_impl should return the number of bytes actually written. raw_write_impl may write less than len bytes, but it should always write at least one byte on success. raw_write_impl should return as soon as possible after at least one byte of data is written.

    TextReader

    pub(open) trait TextReader {
    async fn read(Self) -> String?
    async fn read_all(Self) -> String
    async fn read_until(Self, StringView, include_delimiter? : Bool) -> String?
    }

    TextReader is the trait for an asynchronous text input stream. TextReader implementations are usually UTF-8 wrappers over BinaryReader.

    read() fetch and returns a chunk of text from the stream. The length of the returned chunk is random. read() will return as soon as enough data has arrived.

    read_all() returns all remaining text in the stream.

    read_until() is similar to BinaryReader::read_until, except that it operates over text instead of binary input.

    TextWriter

    pub(open) trait TextWriter {
    async fn write(Self, StringView) -> Unit
    async fn write_json(Self,
    Json
    ) -> Unit
    }

    TextWriter is the trait for an asynchronous text output stream. TextWriter implementations are usually UTF-8 wrappers over BinaryWriter.

    write() writes a piece of text to the underlying stream.

    write_json() writes the text representation of a JSON object to the underlying stream.

    PipeClosed

    This error will be raised when writing to a pipe whose read end is already closed.

    ReaderClosed

    Error raised by binary reader APIs when the binary stream reaches EOF before enough data can be fetched according to user request

    BinaryReaderController

    type BinaryReaderController

    BinaryReaderController::BinaryReaderController

    fn BinaryReaderController::BinaryReaderController() -> BinaryReaderController

    Create a new binary reader controller

    BinaryWriterController

    type BinaryWriterController

    BinaryWriterController::BinaryWriterController

    fn BinaryWriterController::BinaryWriterController() -> BinaryWriterController

    Create a new binary writer controller

    BufferedWriter

    type BufferedWriter[W]

    A buffered writer with a fixed size buffer that wraps around a normal writer. Data will be buffered, instead of being comitted to the underlying writer immediately, unless the buffer is full or explicit flush is requested. Buffering can can improve performance for network connections etc. where small writes are expensive.

    Always remember to call flush before writing directly to the underlying writer. Otherwise some data may remain uncommitted.

    BufferedWriter::capacity

    fn[W] BufferedWriter::capacity(self : BufferedWriter[W]) -> Int

    Return the internal buffer size of a buffered writer.

    BufferedWriter::flush

    async fn[W : BinaryWriter + BinaryWriterImpl] BufferedWriter::flush(self : BufferedWriter[W]) -> Unit

    Flush all buffered data and commit them to the underlying writer.

    BufferedWriter::new

    fn[W] BufferedWriter::new(writer : W, size? : Int) -> BufferedWriter[W]

    Create a buffered writer with fixed buffer size size.

    BufferedWriter::write

    async fn[W : BinaryWriter + BinaryWriterImpl] BufferedWriter::write(self : BufferedWriter[W], data : BytesView) -> Unit

    BufferedWriter::write_reader

    async fn[W : BinaryWriter + BinaryWriterImpl] BufferedWriter::write_reader(self : BufferedWriter[W], reader : &BinaryReader) -> Unit

    MemoryReader

    type MemoryReader

    A reader for transferring data in-memory. This is useful for passing in-memory to reader-receiving API.

    MemoryReader::MemoryReader

    #callsite(autofill(loc))
    fn MemoryReader::MemoryReader(f : async (&BinaryWriter) -> Unit, loc~ : SourceLoc) -> MemoryReader

    Create a in-memory reader by providing a data-writting function. The callback receive a auxiliary writer as argument, data written to the writer can be read from the in-memory reader.

    The callback function will be run in a background task. the reader will reach EOF hen the callback function terminates normally.

    If the callback function failed, all read operation on the in-memory reader will fail with the same error.

    If the reader is closed before the callback completes, the callback will be cancelled automatically.

    MemoryReader::close

    fn MemoryReader::close(self : MemoryReader) -> Unit

    Close the reader. If the writer callback is still running, it will be cancelled.

    MemoryReader::drop_all

    async fn MemoryReader::drop_all(self : MemoryReader) -> Int64

    MemoryReader::drop_exactly

    async fn MemoryReader::drop_exactly(self : MemoryReader, len : Int) -> Unit

    MemoryReader::peek

    async fn MemoryReader::peek(self : MemoryReader) -> BytesView?

    MemoryReader::peek_at_least

    async fn MemoryReader::peek_at_least(self : MemoryReader, min_len : Int) -> BytesView

    MemoryReader::peek_exactly

    async fn MemoryReader::peek_exactly(self : MemoryReader, len : Int) -> BytesView

    MemoryReader::peek_until

    async fn MemoryReader::peek_until(self : MemoryReader, pattern : BytesView, include_delimiter? : Bool) -> BytesView?

    MemoryReader::read

    async fn MemoryReader::read(self : MemoryReader) -> Bytes?

    MemoryReader::read_all

    async fn MemoryReader::read_all(self : MemoryReader) -> Bytes

    MemoryReader::read_at_least

    async fn MemoryReader::read_at_least(self : MemoryReader, min_len : Int) -> Bytes

    MemoryReader::read_exactly

    async fn MemoryReader::read_exactly(self : MemoryReader, len : Int) -> Bytes

    MemoryReader::read_into

    async fn MemoryReader::read_into(self : MemoryReader, dst : FixedArray[Byte], offset? : Int, max_len? : Int) -> Int

    MemoryReader::read_until

    async fn MemoryReader::read_until(self : MemoryReader, pattern : BytesView, include_delimiter? : Bool) -> Bytes?

    PipeRead

    type PipeRead

    The read end of a pipe. Data written to the write end can be read from here.

    PipeRead::close

    fn PipeRead::close(self : PipeRead) -> Unit

    Close the read end of a pipe. After closing the read end, writing to the write end result in error.

    PipeRead::drop_all

    async fn PipeRead::drop_all(self : PipeRead) -> Int64

    PipeRead::drop_exactly

    async fn PipeRead::drop_exactly(self : PipeRead, len : Int) -> Unit

    PipeRead::peek

    async fn PipeRead::peek(self : PipeRead) -> BytesView?

    PipeRead::peek_at_least

    async fn PipeRead::peek_at_least(self : PipeRead, min_len : Int) -> BytesView

    PipeRead::peek_exactly

    async fn PipeRead::peek_exactly(self : PipeRead, len : Int) -> BytesView

    PipeRead::peek_until

    async fn PipeRead::peek_until(self : PipeRead, pattern : BytesView, include_delimiter? : Bool) -> BytesView?

    PipeRead::read

    async fn PipeRead::read(self : PipeRead) -> Bytes?

    PipeRead::read_all

    async fn PipeRead::read_all(self : PipeRead) -> Bytes

    PipeRead::read_at_least

    async fn PipeRead::read_at_least(self : PipeRead, min_len : Int) -> Bytes

    PipeRead::read_exactly

    async fn PipeRead::read_exactly(self : PipeRead, len : Int) -> Bytes

    PipeRead::read_into

    async fn PipeRead::read_into(self : PipeRead, dst : FixedArray[Byte], offset? : Int, max_len? : Int) -> Int

    PipeRead::read_until

    async fn PipeRead::read_until(self : PipeRead, pattern : BytesView, include_delimiter? : Bool) -> Bytes?

    PipeWrite

    type PipeWrite

    The write end of a pipe. Data written here can be read from the read end.

    PipeWrite::close

    fn PipeWrite::close(self : PipeWrite) -> Unit

    Close the write end of a pipe. After closing the write end, reading from the read end result in EOF.

    PipeWrite::write

    async fn PipeWrite::write(self : PipeWrite, data : BytesView) -> Unit

    PipeWrite::write_reader

    async fn PipeWrite::write_reader(self : PipeWrite, reader : &BinaryReader) -> Unit

    Utf8Reader

    type Utf8Reader

    A UTF-8 reader wrapper, converting a binary stream to a text stream via UTF-8 decoding.

    Utf8Reader::Utf8Reader

    fn[R : BinaryReader + BinaryReaderImpl] Utf8Reader::Utf8Reader(inner : R) -> Utf8Reader

    Create a new UTF-8 text reader from an underlying binary reader

    Utf8Reader::read

    async fn Utf8Reader::read(self : Utf8Reader) -> String?

    Utf8Reader::read_all

    async fn Utf8Reader::read_all(self : Utf8Reader) -> String

    Utf8Reader::read_until

    async fn Utf8Reader::read_until(self : Utf8Reader, pattern : StringView, include_delimiter? : Bool) -> String?

    Utf8TextView

    type Utf8TextView

    A bidirectional UTF-8 text view over a binary IO endpoint. Supports both text reading and writing. This type can be used to provide a builtin text view for a binary IO endpoint, and is often exposed via a .text field on the underlying object.

    Utf8TextView::read

    async fn Utf8TextView::read(self : Utf8TextView) -> String?

    Utf8TextView::read_all

    async fn Utf8TextView::read_all(self : Utf8TextView) -> String

    Utf8TextView::read_until

    async fn Utf8TextView::read_until(self : Utf8TextView, pattern : StringView, include_delimiter? : Bool) -> String?

    Utf8TextView::write

    async fn Utf8TextView::write(self : Utf8TextView, data : StringView) -> Unit

    Utf8TextView::write_json

    async fn Utf8TextView::write_json(self : Utf8TextView, data :
    Json
    ) -> Unit

    Utf8Writer

    type Utf8Writer

    A UTF-8 writer wrapper, converting a binary stream to a text stream via UTF-8 encoding.

    Utf8Writer::Utf8Writer

    fn[W : BinaryWriter + BinaryWriterImpl] Utf8Writer::Utf8Writer(inner : W) -> Utf8Writer

    Create a new UTF-8 text writer from an underlying binary writer

    Utf8Writer::write

    async fn Utf8Writer::write(self : Utf8Writer, data : StringView) -> Unit

    Utf8Writer::write_json

    async fn Utf8Writer::write_json(self : Utf8Writer, data :
    Json
    ) -> Unit

    pipe

    fn pipe() -> (PipeRead, PipeWrite)

    Create a new in-memory pipe. Data written to the write end can be obtained from the read end. Useful for testing and converting writer-based function to reader.