Compression and archives for MoonBit.
let small = @gzip.gzip(body[:])
let back = @gzip.gunzip(small[:]) // raises if the checksum disagrees
let archive = @zip.zip([@zip.Entry::new("readme.txt", body)][:])
let members = @zip.unzip(archive[:])| Package | What | Specification |
|---|---|---|
| deflate | DEFLATE, both directions, all three block types | RFC 1951 |
| gzip | The gzip container, with CRC-32 | RFC 1952 |
| zlib | The zlib container, with Adler-32 | RFC 1950 |
| zip | The ZIP archive, with Zip64, descriptors and ZipCrypto | APPNOTE 6.3 |
| tar | ustar and pax, and .tar.gz on top | POSIX pax |
| lz4 | The block and frame formats, with xxHash-32 | lz4 block, frame |
| snappy | The block and framing formats, with CRC-32C | Snappy |
| lzma | The LZMA codec and the .lzma container | LZMA SDK |
| xz | The xz container over LZMA2 | xz file format |
| bzip2 | Burrows-Wheeler, move-to-front and Huffman, both directions | bzip2 |
| sevenz | The 7z container: folders of coders, solid or separate | 7z format |
| zstd | Frames, blocks, FSE and Huffman | RFC 8878 |
| brotli | Meta-blocks, contexts, block types and the 122 KB dictionary | RFC 7932 |
@deflate.deflate(body[:]) // level 6, a 32 KiB window
@deflate.deflate(body[:], level=9) // search hardest
@deflate.deflate(body[:], level=0) // store, do not compress
@deflate.deflate(body[:], blocks=Fixed) // pin the block type
@deflate.deflate(body[:], chain=2048) // set the search depth directly
@deflate.inflate(stream[:], limit=1 << 20) // refuse to expand past a megabyte
@gzip.gzip(body[:], name="report.csv", mtime=written_at)
@gzip.gunzip(stream[:], checksum=false) // read what a damaged stream holds| Setting | Default | Why that one |
|---|---|---|
| level | 6 | The usual default; each level maps to the search depths the format's reference tables give it, so a level means the same here as anywhere |
| window | 32768 | The most DEFLATE allows, and the only size a reader can be assumed to handle |
| chain | from level | For a caller who would rather set the depth than pick a level |
| blocks | Auto | The writer measures a dynamic candidate, a fixed one and a stored one per block and keeps the smallest |
| limit | unbounded | What every library does. A reader handling input from elsewhere should set it: a compressed stream says nothing about how far it expands |
| mtime | 0 | So the same input gives the same bytes. A compressor that stamped the clock would make a build unreproducible |
| checksum, length, header_crc | on | A container that carries a checksum and does not look at it is not a container. Off is for reading what a damaged stream still holds |
// out
let writer = @gzip.Writer::new(level=9)
for part in parts {
for b in writer.push(part) { out.push(b) } // maybe nothing, at first
}
for b in writer.finish() { out.push(b) } // the rest, plus the trailer
// back
let reader = @gzip.Reader::new()
for part in chunks {
for b in reader.push(part) { out.push(b) } // may cross a symbol
}
for b in reader.finish() { out.push(b) } // and the trailer is checked herelet archive = @zip.zip([
@zip.Entry::new("hello.txt", b"hello"),
@zip.Entry::new("dir/nested.txt", body, mtime=written_at),
][:])
@zip.names(archive[:]) // list it, without expanding anything
@zip.entry(archive[:], "hello.txt") // one member, expanding only that one
@zip.unzip(archive[:], limit=limit) // all of them, bounded@zip.zip(entries[:], zip64=true) // sixty-four-bit sizes and offsets
@zip.zip(entries[:], descriptor=true) // sizes after each member, not before
@zip.zip(entries[:], password="hunter2") // ZipCrypto
@zip.unzip(archive[:], password="hunter2")@tar.tar([@tar.Entry::new("a.txt", body), @tar.Entry::directory("empty")][:])
@tar.targz(files[:]) // and untargz@zip.unzip(archive[:], codecs=mine[:])@lzma.lzma(body[:]) // the .lzma alone container
@lzma.unlzma(packed[:]) // both a stated length and an end marker
@xz.xz(body[:]) // the container, one block, CRC-64
@xz.unxz(packed[:]) // header, block header, index and footer checked@bzip2.bzip2(body[:], level=9) // 900k blocks
@bzip2.bunzip2(packed[:]) // block CRCs and the combined one checked@sevenz.sevenz(members[:], how=@sevenz.Method::Lzma2, solid=true)
@sevenz.un7z(archive[:]) // folder, substream and signature checksums verified
@sevenz.names(archive[:]) // the directory only, without touching the data@zstd.zstd(body[:], level=3)
@zstd.unzstd(packed[:])@brotli.brotli(body[:]) // window is a parameter, 10 to 24
@brotli.unbrotli(packed[:], limit=1 << 24)| Format | Bytes | Of the original |
|---|---|---|
| gzip | 1,983 | 1.1% |
| lz4 block | 1,997 | 1.1% |
| lz4 frame (three 64 KiB blocks) | 4,576 | 2.5% |
| deflate, dynamic blocks | 2,661 | 1.4% |
| zlib | 2,667 | 1.4% |
| snappy block | 9,884 | 5.4% |
| snappy frame (three 64 KiB chunks) | 12,439 | 6.8% |
| bzip2 | 853 | 0.5% |
| xz | 468 | 0.3% |
| lzma, alone | 415 | 0.2% |
| zstd | 3,042 | 1.7% |
| brotli | 682 | 0.4% |
| — nothing | 184,000 | 100% |
moon add moonbitstack/moonzippub(all) struct Adler {
a : UInt
b : UInt
}pub(all) struct Castagnoli {
state : UInt
}pub(all) struct Crc {
state : UInt
}fn adler32(data : BytesView) -> UIntfn crc32(data : BytesView) -> UIntfn crc32c(data : BytesView) -> UIntfn mask_crc32c(sum : UInt) -> UInt