#gaato/brotli

moon add gaato/brotli

///|
test "decompress a whole stream" {
  let compressed = b"\x0b\x02\x80hello\x03" // brotli.compress(b"hello")
  let text = @brotli.decompress(compressed, max_output=16 * 1024 * 1024)
  assert_eq(text, b"hello")
}

///|
test "compress and decompress" {
  let text = b"hello, hello, hello, brotli"
  let compressed = @brotli.compress(text)
  assert_eq(@brotli.decompress(compressed), text)
}

///|
fn decode_all(chunks : Array[Bytes], sink : (BytesView) -> Unit) -> Unit raise {
  let decoder = @brotli.Decoder()
  let out = FixedArray::make(64 * 1024, b'\x00')
  for i, chunk in chunks {
    let mut input = chunk[:]
    for ;; {
      let status = decoder.step_into(input, out, end=i == chunks.length() - 1)
      input = input[decoder.last_consumed():]
      sink(Bytes::from_array(out[:decoder.last_produced()]))
      match status {
        Done => return
        NeedMoreInput => break
        NeedMoreOutput => ()
      }
    }
  }
}

///|
test "decode a stream chunk by chunk" {
  let decoded = Buffer()
  decode_all([b"\x0b\x02", b"\x80hel", b"lo\x03"], bytes => {
    decoded.write_bytes(bytes.to_owned())
  })
  assert_eq(decoded.to_bytes(), b"hello")
}

///|
fn encode_parts(parts : Array[Bytes], sink : (BytesView) -> Unit) -> Unit {
  let encoder = @brotli.Encoder()
  let out = FixedArray::make(64 * 1024, b'\x00')
  for i, part in parts {
    let action : @brotli.EncodeAction = if i == parts.length() - 1 {
      Finish
    } else {
      Flush
    }
    let mut input = part[:]
    for ;; {
      let status = encoder.step_into(input, out, action~)
      input = input[encoder.last_consumed():]
      sink(Bytes::from_array(out[:encoder.last_produced()]))
      if status is (Done | NeedMoreInput) {
        break
      }
    }
  }
}

///|
test "encode a stream part by part" {
  let encoded = Buffer()
  encode_parts([b"<html>", b"<body>hello</body>", b"</html>"], bytes => {
    encoded.write_bytes(bytes.to_owned())
  })
  let decoded = @brotli.decompress(encoded.to_bytes())
  assert_eq(decoded, b"<html><body>hello</body></html>")
}

///|
test "stop at the end of the stream" {
  let input = b"\x0b\x02\x80hello\x03" + b"what follows"
  let decoder = @brotli.Decoder(until=EndOfMember)
  let out = FixedArray::make(64, b'\x00')
  let status = decoder.step_into(input, out, end=false)
  assert_eq(status, Done)
  assert_eq(Bytes::from_array(out[:decoder.last_produced()]), b"hello")
  assert_eq(input[decoder.last_consumed():], b"what follows")
}

#What it does

• Decodes every RFC 7932 stream, including the static dictionary and its 121 transforms.
• Takes input in chunks of any size, down to one byte, and writes output into buffers of any size, under the rules of gaato/codec_bytes.
• Memory is the window (at most 16 MiB, and no more than the output so far) plus tables of at most 1.3 MiB; memory_usage() reports it. Decoder(max_window_bits=n) refuses streams that need a window over 2^n - 16 bytes with LimitExceeded, saying how much the stream needs.
• Every malformed stream raises CodecError(kind, message) with Truncated, Corrupt or TrailingData (bytes after the stream); decompress(max_output=n) raises LimitExceeded past n bytes. It does not abort. A step that raises first hands out the output that comes before the error (answering NeedMoreOutput while it does not fit), and last_consumed() and last_produced() count what that step took and wrote: a corrupt stream gives the same output, then the same error, however it is split. On TrailingData, last_consumed() stops at the end of the stream.
• Encodes with greedy LZ77 (lazy by one byte) and one prefix code per meta-block: about google/brotli's quality 4, and close to gzip -6. Its streams decode with google/brotli's decoder (scripts/crosscheck.mbtx).
• The encoder's memory is its window (Encoder(window_bits=n), 22 by default: 4 MiB, growing with the input to at most twice what it has taken, and up to a quarter more before it slides) plus a 64 KiB meta-block, its output and about 1 MiB of fixed tables.
• The rules the decoder applies to what it reads (the window, the meta-block lengths, block switches, distances and the inverse move-to-front), where dictionary words are written, and the canonical codes the encoder writes are proved against RFC 7932 with moon prove; PROOFS.md lists what is proved and what is trusted.

#License

#gaato/brotli

moon add gaato/brotli

///|
test "decompress a whole stream" {
  let compressed = b"\x0b\x02\x80hello\x03" // brotli.compress(b"hello")
  let text = @brotli.decompress(compressed, max_output=16 * 1024 * 1024)
  assert_eq(text, b"hello")
}

///|
test "compress and decompress" {
  let text = b"hello, hello, hello, brotli"
  let compressed = @brotli.compress(text)
  assert_eq(@brotli.decompress(compressed), text)
}

///|
fn decode_all(chunks : Array[Bytes], sink : (BytesView) -> Unit) -> Unit raise {
  let decoder = @brotli.Decoder()
  let out = FixedArray::make(64 * 1024, b'\x00')
  for i, chunk in chunks {
    let mut input = chunk[:]
    for ;; {
      let status = decoder.step_into(input, out, end=i == chunks.length() - 1)
      input = input[decoder.last_consumed():]
      sink(Bytes::from_array(out[:decoder.last_produced()]))
      match status {
        Done => return
        NeedMoreInput => break
        NeedMoreOutput => ()
      }
    }
  }
}

///|
test "decode a stream chunk by chunk" {
  let decoded = Buffer()
  decode_all([b"\x0b\x02", b"\x80hel", b"lo\x03"], bytes => {
    decoded.write_bytes(bytes.to_owned())
  })
  assert_eq(decoded.to_bytes(), b"hello")
}

///|
fn encode_parts(parts : Array[Bytes], sink : (BytesView) -> Unit) -> Unit {
  let encoder = @brotli.Encoder()
  let out = FixedArray::make(64 * 1024, b'\x00')
  for i, part in parts {
    let action : @brotli.EncodeAction = if i == parts.length() - 1 {
      Finish
    } else {
      Flush
    }
    let mut input = part[:]
    for ;; {
      let status = encoder.step_into(input, out, action~)
      input = input[encoder.last_consumed():]
      sink(Bytes::from_array(out[:encoder.last_produced()]))
      if status is (Done | NeedMoreInput) {
        break
      }
    }
  }
}

///|
test "encode a stream part by part" {
  let encoded = Buffer()
  encode_parts([b"<html>", b"<body>hello</body>", b"</html>"], bytes => {
    encoded.write_bytes(bytes.to_owned())
  })
  let decoded = @brotli.decompress(encoded.to_bytes())
  assert_eq(decoded, b"<html><body>hello</body></html>")
}

///|
test "stop at the end of the stream" {
  let input = b"\x0b\x02\x80hello\x03" + b"what follows"
  let decoder = @brotli.Decoder(until=EndOfMember)
  let out = FixedArray::make(64, b'\x00')
  let status = decoder.step_into(input, out, end=false)
  assert_eq(status, Done)
  assert_eq(Bytes::from_array(out[:decoder.last_produced()]), b"hello")
  assert_eq(input[decoder.last_consumed():], b"what follows")
}

#What it does

• Decodes every RFC 7932 stream, including the static dictionary and its 121 transforms.
• Takes input in chunks of any size, down to one byte, and writes output into buffers of any size, under the rules of gaato/codec_bytes.
• Memory is the window (at most 16 MiB, and no more than the output so far) plus tables of at most 1.3 MiB; memory_usage() reports it. Decoder(max_window_bits=n) refuses streams that need a window over 2^n - 16 bytes with LimitExceeded, saying how much the stream needs.
• Every malformed stream raises CodecError(kind, message) with Truncated, Corrupt or TrailingData (bytes after the stream); decompress(max_output=n) raises LimitExceeded past n bytes. It does not abort. A step that raises first hands out the output that comes before the error (answering NeedMoreOutput while it does not fit), and last_consumed() and last_produced() count what that step took and wrote: a corrupt stream gives the same output, then the same error, however it is split. On TrailingData, last_consumed() stops at the end of the stream.
• Encodes with greedy LZ77 (lazy by one byte) and one prefix code per meta-block: about google/brotli's quality 4, and close to gzip -6. Its streams decode with google/brotli's decoder (scripts/crosscheck.mbtx).
• The encoder's memory is its window (Encoder(window_bits=n), 22 by default: 4 MiB, growing with the input to at most twice what it has taken, and up to a quarter more before it slides) plus a 64 KiB meta-block, its output and about 1 MiB of fixed tables.
• The rules the decoder applies to what it reads (the window, the meta-block lengths, block switches, distances and the inverse move-to-front), where dictionary words are written, and the canonical codes the encoder writes are proved against RFC 7932 with moon prove; PROOFS.md lists what is proved and what is trusted.

#License