zstd

    A streaming Zstandard (RFC 8878) decoder in pure MoonBit: arbitrary input chunks, bounded output slices, memory bounded by the window.

    zstd
    zstandard
    compression
    decompression
    rfc8878
    streaming
    Download zip
    Author
    Version
    0.5.1
    License
    Apache-2.0
    Last updated
    yesterday
    Downloads
    3K

    #gaato/zstd

    A streaming Zstandard (RFC 8878) decoder and encoder in pure MoonBit, for Native, JS, Wasm and Wasm GC. Both take input in chunks of any size and write output into slices of any size; the decoder keeps its memory within the window you allow, so it fits an HTTP client's zstd content coding as well as a whole file, and the encoder compresses about as well as zstd at the same level.

    #Install

    moon add gaato/zstd

    #Usage

    One shot:

    ///|
    test "decompress a whole stream" {
    let compressed = b"\x28\xb5\x2f\xfd\x20\x05\x29\x00\x00\x68\x65\x6c\x6c\x6f" // zstd.compress(b"hello")
    let text = @zstd.decompress(compressed, max_output=16 * 1024 * 1024)
    assert_eq(text, b"hello")
    }

    ///|
    test "compress and decompress" {
    let text = b"hello, hello, hello, zstd"
    let compressed = @zstd.compress(text) // level 3, as zstd's default
    assert_eq(@zstd.decompress(compressed), text)
    let smaller = @zstd.compress(text, level=12)
    assert_eq(@zstd.decompress(smaller), text)
    }

    Streaming: feed input as it arrives, take output in slices of the size you choose. A stream may hold several frames and nothing marks the last one, so every step says whether its input is the last: end=false before, end=true with the last input. Status is gaato/codec_bytes's, re-exported: match Done, NeedMoreInput and NeedMoreOutput unqualified, or write @zstd.Status::Done.

    ///|
    fn decode_all(chunks : Array[Bytes], sink : (BytesView) -> Unit) -> Unit raise {
    let decoder = @zstd.Decoder(max_window_bits=23) // 8 MiB, as RFC 9659 asks of HTTP
    let out = FixedArray::make(64 * 1024, b'\x00')
    for i, chunk in chunks {
    let mut input = chunk[:]
    for ;; {
    let status = decoder.step_into(input, out, end=i == chunks.length() - 1)
    input = input[decoder.last_consumed():]
    sink(Bytes::from_array(out[:decoder.last_produced()]))
    match status {
    Done => return
    NeedMoreInput => break
    NeedMoreOutput => ()
    }
    }
    }
    }

    ///|
    test "decode a stream chunk by chunk" {
    let decoded = Buffer()
    decode_all(
    [b"\x28\xb5\x2f", b"\xfd\x20\x05\x29\x00\x00\x68", b"\x65\x6c\x6c\x6f"],
    bytes => decoded.write_bytes(bytes.to_owned()),
    )
    assert_eq(decoded.to_bytes(), b"hello")
    }

    A frame followed by something else: until=EndOfMember stops right after the first frame and consumes nothing past it, so what follows is input[last_consumed():]. A skippable frame counts as a frame here.

    ///|
    test "stop after the first frame" {
    let frame = b"\x28\xb5\x2f\xfd\x20\x05\x29\x00\x00\x68\x65\x6c\x6c\x6f"
    let input = Bytes::from_array([..frame[:], ..b"rest"])
    let decoder = @zstd.Decoder(until=EndOfMember)
    let out = FixedArray::make(64, b'\x00')
    let status = decoder.step_into(input, out, end=true)
    assert_true(status is Done)
    assert_eq(Bytes::from_array(out[:decoder.last_produced()]), b"hello")
    assert_eq(input[decoder.last_consumed():], b"rest")
    }

    Encoding works the same way. Flush makes the output so far decode to the input so far, say to send one part of an HTTP response; Finish ends the frame. A flush ends the block in progress, 3 bytes of block header besides what it holds; a flush with nothing new since the last writes nothing, and so does one before any input, since the frame header comes with the first block.

    ///|
    fn encode_parts(parts : Array[Bytes], sink : (BytesView) -> Unit) -> Unit raise {
    let encoder = @zstd.Encoder()
    let out = FixedArray::make(64 * 1024, b'\x00')
    for i, part in parts {
    let action : @zstd.EncodeAction = if i == parts.length() - 1 {
    Finish
    } else {
    Flush
    }
    let mut input = part[:]
    for ;; {
    let status = encoder.step_into(input, out, action~)
    input = input[encoder.last_consumed():]
    sink(Bytes::from_array(out[:encoder.last_produced()]))
    if status is (Done | NeedMoreInput) {
    break
    }
    }
    }
    }

    ///|
    test "encode a stream part by part" {
    let encoded = Buffer()
    encode_parts([b"<html>", b"<body>hello</body>", b"</html>"], bytes => {
    encoded.write_bytes(bytes.to_owned())
    })
    let decoded = @zstd.decompress(encoded.to_bytes())
    assert_eq(decoded, b"<html><body>hello</body></html>")
    }

    Decoder and Encoder implement gaato/codec_bytes's Decode and Encode, so code written against that contract (an HTTP client, a command-line tool) takes them as it takes any other codec there.

    #What it decodes

    Every RFC 8878 frame without a dictionary: raw, RLE and compressed blocks, Huffman-coded literals (one or four streams, with or without a new tree), every FSE table mode, and the content checksum (XXH64). A stream may hold several frames, and skippable frames among them, as the zstd tool writes them.

    Input comes in chunks of any size, down to one byte, and output goes into slices of any size, under the rules of gaato/codec_bytes. The header decisions and the Huffman decoding tables are proved against RFC 8878; PROOFS.md lists what is proved and what is trusted.

    A dictionary, given to Decoder(dictionary=d) or decompress(dictionary=d), comes as if just before every frame's output, so a frame made with zstd -D decodes, and so does RFC 9842's dcz coding. Dictionary::raw(bytes) takes the bytes as they are, as dcz does; Dictionary::parse(bytes) reads a formatted dictionary, as zstd --train writes them, whose entropy tables and repeat offsets the frames start from, and takes anything else as raw content, as zstd -D does. A frame that names another dictionary raises DictionaryMismatch, and one that names a dictionary when none is given DictionaryRequired(Some(id)).

    #Reading frames without decoding them

    inspect_frame reads the header of the frame at the start of some bytes: a Zstandard frame's window, content size, dictionary and checksum flag, or a skippable frame's variant and size. frame_size walks the frame's block headers to where it ends, without decoding it, as zstd's ZSTD_findFrameCompressedSize does, so a stream can be cut into frames, or checked to be whole, cheaply. skippable_frame writes a skippable frame, which carries bytes of your own beside Zstandard frames and which every decoder passes over (RFC 9842's dcz coding starts with one), and read_skippable_frame reads one back.

    ///|
    test "frames, read without decoding them" {
    let note = @zstd.skippable_frame(3, b"made by me")
    let frame = @zstd.compress(b"hello, hello, hello")
    let stream = Bytes::from_array([..note[:], ..frame[:]])
    assert_eq(@zstd.frame_size(stream), 18)
    let rest = stream[18:]
    guard @zstd.inspect_frame(rest) is Zstd(info) else { fail("not a frame") }
    assert_eq(info.content_size, Some(19))
    assert_eq(@zstd.read_skippable_frame(stream).1, b"made by me")
    assert_eq(@zstd.decompress(stream), b"hello, hello, hello")
    }

    #Encoding

    Encoder(level=n) and compress(level=n) take zstd's levels 1 to 12, 3 by default as in zstd (0 is the default too), with zstd 1.5.7's parameters for each level and input size, and its negative levels, -1 down to -131072, the faster ones zstd --fast=N makes as level -N: they skip ahead faster the lower they go and leave literals uncoded. For large inputs, and those of unknown size, that is fast at 1 and 2, double-fast at 3 and 4, and greedy, lazy and lazy2 from 5 to 12; a smaller input can get another strategy, as in zstd. On the Silesia and Canterbury corpora, the totals come within 1 % of zstd -N's at every level. The frame carries the XXH64 content checksum unless checksum=false. Its bytes do not depend on how the input or the output is split: only on the input, the options and where Flush falls.

    compress and Encoder(content_size=n) write the size in the frame header and pick the parameters for it, as zstd does: a small input gets small tables and declares a window no larger than itself.

    Encoder(dictionary=d) and compress(dictionary=d) put a Dictionary before the frame as zstd -D does: the frame's matches may reach into it until the input is a window past it, and the parameters are chosen for the input and the dictionary together. With a formatted dictionary, the frame names it by its id, and its first block may code literals and sequences with the dictionary's tables and start from its repeat offsets. The frames decode with zstd -d -D, and on small inputs come out the size of zstd's or smaller.

    Not included: levels 13 to 22 (they act as 12), long-distance matching, multithreading, and training dictionaries (zstd --train makes them).

    #Memory

    Memory is the window (at most the frame's Window_Size, and no more than its output so far), one compressed block of at most 128 KiB and its literals, plus 13 KiB of tables. Decoder(max_window_bits=n) refuses frames that need a window over 2^n bytes, before allocating anything; the default 27 (128 MiB) is the reference decoder's. For HTTP, RFC 9659 asks for 23 (8 MiB). memory_usage() reports all of it: at most 2^n bytes plus 256 KiB plus 13,156 bytes of tables, which decoder_memory_bound(max_window_bits=n) gives before you make a decoder.

    The encoder's memory is set by its level: for input of unknown size, about 3.1 MiB at level 1, 6.8 MiB at level 3 and 50 MiB at level 12 (its 4 MiB window, as much again before it slides, and 40 MiB of match tables). A known content_size shrinks the window and the tables to fit. memory_usage() reports it, and stays within encoder_memory_bound(level~, content_size?, dictionary_len~) whatever the steps. compress_bound(n) is the most compress writes for n bytes, for sizing an output buffer or a limit.

    #Errors

    Every malformed stream raises gaato/codec's CodecError(kind, message); the decoder does not abort. The kinds:

    • Truncated: the input ends inside a frame.
    • Corrupt: the input is not a valid stream.
    • TrailingData: bytes after the last frame that start no frame.
    • ChecksumMismatch: the content checksum does not match.
    • LimitExceeded: a window over max_window_bits, with resource: Memory; more than max_output from decompress, with resource: Output.
    • DictionaryRequired: the frame names a dictionary, with its id.

    The encoder raises only when it was given content_size and the input does not match it: ChecksumMismatch, once the input passes the size (the output for the input up to it is handed out first) or at Finish when it falls short. The error sticks until reset.

    In a catch on these functions, CodecError(kind, _) and the kinds match unqualified, with no import; to name the types elsewhere, import gaato/codec_bytes, which re-exports them.

    A step that raises first hands out the output that comes before the error (answering NeedMoreOutput while it does not fit), and last_consumed() and last_produced() count what that step took and wrote: a corrupt stream gives the same output, then the same error, however it is split. On TrailingData, last_consumed() stops at the end of the stream.

    #When to use it

    gaato/zstd fits zstd that arrives in pieces or comes from anyone. Decoder takes input in pieces down to one byte, writes into your slices and holds the window you allow, 128 MiB by default (a process decoding 256 MiB of output in 64 KiB pieces peaks at 15.9 MiB, most of it the runtime and the compressed input); it allocates nothing from a size a header claims. It agreed with zstd -d on all 36 valid and 318 invalid streams of the comparison, checks the content checksum, and its frame headers and Huffman tables are proved against RFC 8878. It runs on Native, JS, Wasm and Wasm GC and takes no dictionary. Its encoder, at each level from 1 to 12, compresses the corpora to within 1 % of zstd, and its frames never depend on how the input arrives.

    Milky2018/zstd, which also compresses and takes dictionaries, takes about 4.1 (Native), 5.1 (Wasm), 4.0 (Wasm GC) and 2.7 (JS) times as long as gaato/zstd on the benchmarks' text. Its encoder writes the input uncompressed at its default level 3; at level 22 it writes twice what gaato/zstd writes at level 12 on that text, and takes 9 (Wasm GC) to 16 (JS) times as long.

    #License

    Apache-2.0. The test frames under testdata/upstream come from facebook/zstd under its BSD license; see NOTICE.

    EncodeAction

    What Encoder::step does once its input is taken: Continue, Flush or Finish, as every gaato/codec_bytes encoder takes it.

    Status

    using @gaato/codec_bytes { type Status }

    Outcome of one Decoder::step or Encoder::step: Done, NeedMoreInput or NeedMoreOutput, as every gaato/codec_bytes codec answers.

    Decoder

    pub struct Decoder {
    // private fields
    }

    A streaming Zstandard decoder (RFC 8878). Feed it input in chunks of any size with step and collect output in slices of any size; it keeps the window (no more than a frame's Window_Size, and no more than its output so far), one compressed block and fixed-size tables, never the whole input or output.
    impl Decode for Decoder

    Decoder::Decoder

    fn Decoder::Decoder(max_window_bits? : Int, until? :
    Until
    , dictionary? : Dictionary) -> Decoder

    A decoder at the start of a stream. It refuses, with LimitExceeded({ resource: Memory, .. }), a frame whose Window_Size exceeds 1 << max_window_bits bytes (the window memory it would need). The default 27 is the reference decoder's default limit; RFC 9659 asks HTTP clients to accept 23 (8 MiB). Values are clamped to 10..30.

    until says where the stream ends. EndOfInput, the default, decodes every frame up to the end of the input. EndOfMember stops right after the first frame, a skippable frame included, and consumes nothing past it, so that input[last_consumed():] is what follows.

    dictionary is used by every frame that names it, or names none (RFC 8878 section 5): its content comes as if just before the frame's output, and a match may reach into it however long the frame is, as zstd's decoder lets it. A frame that names another dictionary raises DictionaryMismatch; without dictionary, a frame that names one raises DictionaryRequired(Some(id)). The decoder keeps the dictionary by reference, so the bytes it is a view of stay alive as long as the decoder, and memory_usage does not count them.

    Decoder::is_finished

    fn Decoder::is_finished(self : Decoder) -> Bool

    Whether the stream has ended and all its output has been handed out.

    Decoder::last_consumed

    fn Decoder::last_consumed(self : Decoder) -> Int

    Input bytes the latest step accepted: all of them on NeedMoreInput. A step that raised accepted the bytes up to the error; for TrailingData, exactly up to the end of the stream, or none if the stream ended before this step's input.

    Decoder::last_produced

    fn Decoder::last_produced(self : Decoder) -> Int

    Output bytes the latest step wrote, a step that raised included: the output before the error.

    Decoder::memory_usage

    fn Decoder::memory_usage(self : Decoder) -> Int

    Bytes the decoder holds: the window, the compressed block and its literals, which grow with the stream, and the Huffman and FSE tables and their scratch space, 13,156 bytes allocated with the decoder. At most (1 << max_window_bits) + 2 * 128 KiB plus those; the window is released when a frame needs a smaller one, so it may drop between frames. A dictionary is the caller's, kept by reference, and not counted; nor are the frame header being read and the checksum's state, under 100 bytes.

    Decoder::reset

    fn Decoder::reset(self : Decoder) -> Unit

    Starts a new stream with the same options, keeping the tables' memory. The window and the block buffers are released.

    Decoder::step

    fn Decoder::step(self : Decoder, input : BytesView, output : MutArrayView[Byte], end~ : Bool) ->
    Status
    raise
    CodecError

    Decodes from input into output; see last_consumed and last_produced for how much of each it used.

    A stream is one or more frames, and nothing marks its last one, so pass end=true once input holds the last bytes and end=false before: the stream then ends after the frame in progress, and a stream cut short raises Truncated. Without end, the decoder answers NeedMoreInput between frames. Bytes after the last frame that start no frame raise TrailingData. Under until=EndOfMember the stream ends after its first frame, whatever end says.

    An error comes after all the output before it: when output fills first, the step answers NeedMoreOutput and a later one hands out the rest and raises. After an error the decoder raises it again, consuming and writing nothing, until reset.

    Decoder::step_into

    fn Decoder::step_into(self : Decoder, input : BytesView, output : FixedArray[Byte], end~ : Bool) ->
    Status
    raise
    CodecError

    step into a whole FixedArray, with bulk copies.

    Dictionary

    pub struct Dictionary {
    // private fields
    }

    A dictionary for Decoder and Encoder: its content, held by reference (the bytes it is a view of stay alive as long as it does), and its id.

    Dictionary::content

    fn Dictionary::content(self : Dictionary) -> BytesView

    The content: the bytes that come before every frame's output.

    Dictionary::id

    fn Dictionary::id(self : Dictionary) -> UInt

    The id a frame names this dictionary by: 0 for raw content.

    Dictionary::is_formatted

    fn Dictionary::is_formatted(self : Dictionary) -> Bool

    Whether the dictionary is formatted: has entropy tables and repeat offsets of its own.

    Dictionary::parse

    fn Dictionary::parse(data : BytesView) -> Dictionary raise
    CodecError

    A dictionary from data, as zstd -D reads one: a formatted dictionary (RFC 8878 section 5, as zstd --train writes them) if it starts with the magic number 0xEC30A437, raw content otherwise. The content is a view of data.

    Raises Corrupt for a formatted dictionary whose entropy tables do not parse, whose repeat offsets are 0 or longer than its content, or that ends inside them.

    Dictionary::raw

    fn Dictionary::raw(content : BytesView) -> Dictionary

    A dictionary of raw content, content itself: its id is 0, so it is used for frames that name no dictionary, as RFC 8878 section 5 and zstd use one, and RFC 9842's dcz coding does.

    Encoder

    pub struct Encoder {
    // private fields
    }

    A streaming Zstandard encoder (RFC 8878). Feed it input in chunks of any size with step and collect output in slices of any size. It writes one frame: Raw, RLE or Compressed blocks of at most 128 KiB, whichever is smallest, and the XXH64 content checksum unless asked not to.

    Levels 1..12 select fast, double-fast, greedy, lazy or lazy2 from their size-class parameters. Tree rows use lazy2 with the same numeric row.
    impl Encode for Encoder

    Encoder::Encoder

    fn Encoder::Encoder(level? : Int, checksum? : Bool, content_size? : Int64, dictionary? : Dictionary) -> Encoder

    An encoder at the start of a frame.

    level is zstd's compression level: 1 to 12, 3 by default as in zstd, and 0 is the default too, as in zstd. Levels 13 to 22 are not implemented yet and act as 12. The negative levels, -1 down to -131072 (lower ones act as that), are zstd's faster ones (zstd --fast=N is level -N): they skip ahead faster the lower they go, and leave literals uncoded, as zstd does.

    checksum adds the frame's content checksum, the low 32 bits of the XXH64 of the input, as zstd does by default.

    content_size, when given, is written in the frame header as Frame_Content_Size, and the input must then be exactly that long: step raises ChecksumMismatch once it has more, or at Finish when it has less. Like zstd, the encoder then picks its parameters from the size, and a frame that fits its window is written single-segment, so a decoder reserves no more than the content. A negative content_size counts as not given.

    dictionary comes before every frame, as zstd's -D puts it: the frame's matches may reach into it until the input is a window past it, and the parameters are chosen for the input and the dictionary together, as zstd chooses them. A decoder needs the same dictionary. The encoder keeps it by reference and copies its content into its history at the start of every frame, so its memory grows by the dictionary's length, twice: once for the dictionary and once for the room the history keeps beyond it.

    Encoder::is_finished

    fn Encoder::is_finished(self : Encoder) -> Bool

    Whether the frame has ended and all its output has been handed out.

    Encoder::last_consumed

    fn Encoder::last_consumed(self : Encoder) -> Int

    Input bytes the latest step took.

    Encoder::last_produced

    fn Encoder::last_produced(self : Encoder) -> Int

    Output bytes the latest step wrote, a step that raised included.

    Encoder::memory_usage

    fn Encoder::memory_usage(self : Encoder) -> Int

    Bytes the encoder holds in buffers that grow with the stream: the input block in shared history, the output waiting for the caller, match window and hash, sequence storage, and entropy coding scratch. Bounds for each are in RESOURCE_BOUNDS.md. reset keeps every buffer's capacity, so the count does not drop then; it stays within the bound of the encoder's options.

    Encoder::reset

    fn Encoder::reset(self : Encoder) -> Unit

    Starts a new frame with the same options, keeping the memory of its buffers and tables: the next frame's bytes are those a new encoder writes, and memory_usage still counts what is kept.

    Encoder::step

    Compresses from input into output; see last_consumed and last_produced for how much of each it used. Input that was not taken (when the output is full) must be passed again, with the same action.

    action applies once all of input is taken: Continue only takes input, Flush ends the block in progress, so that the output so far decodes to the input so far without ending the frame, and Finish ends the frame. Both stay in effect while the step answers NeedMoreOutput. A Flush with no input since the last one writes nothing. Status: NeedMoreInput once everything is taken and its output handed out (a Flush is then complete), NeedMoreOutput while output waits, Done once the frame has ended and been handed out; after that, steps take no input and answer Done.

    Raises only when the encoder was given content_size and the input does not match it: ChecksumMismatch, once the input passes it (the bytes up to it are taken as with Continue, and the output they decide is handed out first, so that the output before the error does not depend on how input and output are split) or at Finish when it falls short. The error sticks: later steps raise it again, taking and writing nothing, until reset.

    Encoder::step_into

    fn Encoder::step_into(self : Encoder, input : BytesView, output : FixedArray[Byte], action~ :
    EncodeAction
    ) ->
    Status
    raise
    CodecError

    step into a whole FixedArray, with bulk copies.

    Frame

    pub(all) enum Frame {
    Zstd(FrameInfo)
    Skippable(variant~ : Int, size~ : UInt)
    } derive(Eq,
    Debug
    )

    The frame at the start of some bytes, as inspect_frame sees it.

    Frame::equal

    fn Frame::equal(Frame, Frame) -> Bool

    Compares two frames read by inspect_frame, field by field.

    Frame::not_equal

    fn Frame::not_equal(x : Frame, y : Frame) -> Bool

    Compares two frames read by inspect_frame, field by field.

    Frame::to_repr

    Shows the frame as its fields, for tests and messages.

    FrameInfo

    pub struct FrameInfo {
    window_size : UInt64
    content_size : UInt64?
    dictionary_id : UInt
    has_checksum : Bool
    single_segment : Bool
    header_len : Int
    } derive(Eq,
    Debug
    )

    What a Zstandard frame's header says, as inspect_frame reads it.

    FrameInfo::equal

    fn FrameInfo::equal(FrameInfo, FrameInfo) -> Bool

    Compares two frame headers read by inspect_frame, field by field.

    FrameInfo::not_equal

    fn FrameInfo::not_equal(x : FrameInfo, y : FrameInfo) -> Bool

    Compares two frame headers read by inspect_frame, field by field.

    FrameInfo::to_repr

    Shows the frame header as its fields, for tests and messages.

    compress

    fn compress(input : BytesView, level? : Int, checksum? : Bool, dictionary? : Dictionary) -> Bytes

    Compresses input into one whole Zstandard frame. level, checksum and dictionary are as for Encoder; the frame states its content size, and the parameters are chosen for it, so a small input declares a small window (a single-segment frame, whose window is its content).

    compress_bound

    fn compress_bound(n : Int64) -> Int64

    The most bytes compress or an Encoder writes for n bytes of input, at any level and with any dictionary: the input, 3 bytes for each block (a block is never larger than its bytes stored raw, and is at least 1 KiB, the smallest window, unless it is the last), one block for no input, the magic number and the longest frame header (4 + 14, with a 4-byte dictionary id and an 8-byte content size) and the 4-byte checksum. Each Flush that ends a block early adds 3 bytes. Like ZSTD_compressBound, it allows for blocks smaller than compress writes (its blocks are 128 KiB once the input is), so that it holds for an Encoder whose parameters choose a small window. A negative n counts as 0; past 2^62 the result is the largest Int64.

    decoder_memory_bound

    fn decoder_memory_bound(max_window_bits? : Int) -> Int64

    The most Decoder::memory_usage reports for a decoder made with max_window_bits (clamped to 10..30, as Decoder clamps it): the window, a compressed block and its literals, and the 13,156 bytes of tables made with the decoder. A dictionary is the caller's and is not counted.

    decompress

    fn decompress(input : BytesView, max_output? : Int, max_window_bits? : Int, dictionary? : Dictionary) -> Bytes raise
    CodecError

    Decodes a whole Zstandard stream held in memory: one or more frames. max_output caps the decoded size (unlimited by default) and raises LimitExceeded({ resource: Output, .. }) past it; max_window_bits and dictionary are as for Decoder.

    encoder_memory_bound

    fn encoder_memory_bound(level? : Int, content_size? : Int64, dictionary_len? : Int) -> Int64

    The most Encoder::memory_usage reports for an encoder made with these options, whatever the input and however it is split: what the level's parameters (chosen for content_size and a dictionary of dictionary_len bytes, as Encoder chooses them) allocate, with the room its buffers may double into. level is as for Encoder.

    frame_size

    fn frame_size(input : BytesView) -> Int64 raise
    CodecError

    The length of the frame at the start of input, a Zstandard frame or a skippable one, found by reading its header and walking its block headers (an RLE block holds one byte, a raw or compressed block its size) and its checksum, without decoding anything; input[frame_size(input):] is what follows the frame. As zstd's ZSTD_findFrameCompressedSize.

    Raises Truncated if input ends inside the frame, and Corrupt for what inspect_frame refuses and for a block of the reserved type. The blocks' contents are not checked: Decoder does that.

    inspect_frame

    fn inspect_frame(input : BytesView) -> Frame raise
    CodecError

    Reads the header of the frame at the start of input without decoding it: a Zstandard frame's window, content size, dictionary and checksum flag, or a skippable frame's variant and size. Only the header's bytes are read.

    Raises Truncated if input ends inside the header, and Corrupt if it starts with neither magic number or the header's reserved bit is set. A window or a dictionary is not judged here: Decoder judges those.

    read_skippable_frame

    fn read_skippable_frame(input : BytesView) -> (Int, BytesView) raise
    CodecError

    The variant and the data of the skippable frame at the start of input, a view into input. Raises Truncated if input ends inside the frame, and Corrupt if it does not start with a skippable frame.

    skippable_frame

    fn skippable_frame(variant : Int, data : BytesView) -> Bytes

    A skippable frame of variant (its low four bits are used) holding data: the magic number 0x184D2A50 + variant, the length of data in four little-endian bytes, then data. Decoders skip it, so it carries bytes of the caller's beside Zstandard frames, as the header of RFC 9842's dcz coding does.