codec_bytes

    The streaming contract of byte-stream codecs: step through input chunks into bounded output slices, with bounded memory.

    codec
    compression
    streaming
    Download zip
    Author
    Version
    0.1.2
    License
    Apache-2.0
    Last updated
    16 hours ago
    Downloads
    4K

    #gaato/codec_bytes

    The contract of streaming byte-stream codecs in codecs.mbt: a decoder or an encoder is driven step by step, takes input in chunks of any size, writes into output slices of any size, and keeps its memory within the limits it was given. Code written against Decode and Encode works with every codec that implements them, such as gaato/zstd, gaato/brotli and gaato/deflate.

    #Install

    moon add gaato/codec_bytes

    #Usage

    Write the loop that drives a decoder once, against Decode. Each step consumes a prefix of the input and writes a prefix of the output slice; last_consumed() and last_produced() say how much, and the Status says why it stopped. Pass end=true with the last input, so that a stream cut short raises Truncated instead of ending quietly.

    ///|
    /// Decodes `chunks` with any decoder, handing each slice of output to `sink`.
    fn[D : @codec_bytes.Decode] decode_all(
    decoder : D,
    chunks : Array[Bytes],
    sink : (BytesView) -> Unit,
    ) -> Unit raise {
    let out = FixedArray::make(64 * 1024, b'\x00')
    for i, chunk in chunks {
    let mut input = chunk[:]
    for ;; {
    let status = decoder.step(
    input,
    out.mut_view(),
    end=i == chunks.length() - 1,
    )
    input = input[decoder.last_consumed():]
    sink(Bytes::from_array(out[:decoder.last_produced()]))
    match status {
    Done => return
    NeedMoreInput => break
    NeedMoreOutput => ()
    }
    }
    }
    }

    Any codec's Decoder goes in, @zstd.Decoder() or @gzip.Decoder() alike. To show the contract on its own, here is the smallest decoder that keeps it, one that copies its input:

    ///|
    /// A decoder of the identity format: output is input.
    struct Identity {
    mut taken : Int32
    mut written : Int32
    }

    ///|
    impl @codec_bytes.Decode for Identity with fn step(self, input, output, end~) {
    let n = input.length().min(output.length())
    for i in 0..<n {
    output[i] = input[i]
    }
    self.taken = n
    self.written = n
    if n < input.length() {
    NeedMoreOutput
    } else if end {
    Done
    } else {
    NeedMoreInput
    }
    }

    ///|
    impl @codec_bytes.Decode for Identity with fn last_consumed(self) {
    self.taken
    }

    ///|
    impl @codec_bytes.Decode for Identity with fn last_produced(self) {
    self.written
    }

    ///|
    impl @codec_bytes.Decode for Identity with fn reset(self) {
    self.taken = 0
    self.written = 0
    }

    ///|
    impl @codec_bytes.Decode for Identity with fn memory_usage(_self) {
    0
    }

    ///|
    test "drive a decoder through the contract" {
    let decoder : Identity = { taken: 0, written: 0, }
    let decoded = Buffer()
    decode_all(decoder, [b"hel", b"lo"], bytes => {
    decoded.write_bytes(bytes.to_owned())
    })
    assert_eq(decoded.to_bytes(), b"hello")
    }

    Encoders work the same way through Encode, with an EncodeAction in place of end: Continue takes input, Flush makes the output so far decode to the input so far, and Finish ends the stream.

    #Errors

    A step raises gaato/codec's CodecError(kind, message). This package re-exports it with ErrorKind, Limit and Resource, so it is the only import a consumer needs to name them. In a catch on a step, CodecError(kind, message) and the kinds match unqualified; elsewhere, qualify a constructor by its type, as in @codec_bytes.CodecError::CodecError(kind, _).

    A step that raises first writes the output that comes before the error, and last_consumed() and last_produced() count what it took and wrote. Errors stick: later steps raise the same kind until reset.

    #License

    Apache-2.0.

    CodecError

    using @gaato/codec { type CodecError }

    The error every codec raises: gaato/codec's, re-exported so that a caller of a codec names it with this package as its only import. Its constructor matches unqualified where the type is known, as in CodecError(Corrupt, _) in a catch on a codec's step, or qualified by its type, as in @codec_bytes.CodecError::CodecError(kind, _).

    ErrorKind

    using @gaato/codec { type ErrorKind }

    Why a codec stopped with an error: gaato/codec's, re-exported.

    Limit

    using @gaato/codec { type Limit }

    A limit a stream would exceed: gaato/codec's, re-exported.

    Resource

    using @gaato/codec { type Resource }

    What a limit is about: gaato/codec's, re-exported.

    Decode

    pub(open) trait Decode {
    fn step(Self, BytesView, MutArrayView[Byte], end~ : Bool) -> Status raise
    CodecError

    fn last_consumed(Self) -> Int
    fn last_produced(Self) -> Int
    fn reset(Self) -> Unit
    fn memory_usage(Self) -> Int
    }

    A decoder of a byte stream, driven step by step.

    A step consumes a prefix of input and writes a prefix of output (last_consumed and last_produced say how much), and returns why it stopped. end says whether input reaches the end of the data. The rules every decoder keeps are in the repository's CONTRACT.md: among them, a step that raises first writes all the output the input determines before the error (answering NeedMoreOutput if the slice fills first) and reports what it consumed and wrote, errors stick until reset, and the output and the error do not depend on how input and output are split.

    Encode

    pub(open) trait Encode {
    fn step(Self, BytesView, MutArrayView[Byte], action~ : EncodeAction) -> Status raise
    CodecError

    fn last_consumed(Self) -> Int
    fn last_produced(Self) -> Int
    fn reset(Self) -> Unit
    fn memory_usage(Self) -> Int
    }

    An encoder of a byte stream, driven step by step, with the same rules as Decode; action takes the place of end.

    EncodeAction

    pub(all) enum EncodeAction {
    Continue
    Flush
    Finish
    } derive(Eq,
    Debug
    )

    What an encoder's step should do with the input it has.

    EncodeAction::equal

    EncodeAction::equal and EncodeAction::not_equal, the derived comparison.

    EncodeAction::not_equal

    fn EncodeAction::not_equal(x : EncodeAction, y : EncodeAction) -> Bool

    EncodeAction::equal and EncodeAction::not_equal, the derived comparison.

    EncodeAction::to_repr

    EncodeAction::to_repr, the derived debug representation.

    Status

    pub(all) enum Status {
    Done
    NeedMoreInput
    NeedMoreOutput
    } derive(Eq,
    Debug
    )

    Why a step stopped without an error.

    Status::equal

    fn Status::equal(Status, Status) -> Bool

    Status::equal and Status::not_equal, the derived comparison.

    Status::not_equal

    fn Status::not_equal(x : Status, y : Status) -> Bool

    Status::equal and Status::not_equal, the derived comparison.

    Status::to_repr

    Status::to_repr, the derived debug representation.

    Until

    pub(all) enum Until {
    EndOfInput
    EndOfMember
    } derive(Eq,
    Debug
    )

    Where a decoder stops, for formats whose streams can follow one another.

    Until::equal

    fn Until::equal(Until, Until) -> Bool

    Until::equal and Until::not_equal, the derived comparison.

    Until::not_equal

    fn Until::not_equal(x : Until, y : Until) -> Bool

    Until::equal and Until::not_equal, the derived comparison.

    Until::to_repr

    Until::to_repr, the derived debug representation.

    Source Files