Zstandard support

[fabiospampinato]

What can't you do right now?

Zstandard compresses amazingly well, for my use case up to 4x better than what I can get from gzip, and it decompresses perhaps even more amazingly well.

For my use case I kinda have to use Zstandard given its characteristics, but there doesn't seem to be a great port of Zstandard for the browser, plus I'm already using fflate which I find well done and pretty fast, but it just doesn't support Zstandard.

An optimal solution

Optimally fflate would add support for compressing and decompressing (although just decompressing would be enough for me, and I'd guess for most other people too) using Zstandard with a high quality and fast implementation.

(How) is this done by other libraries?

• node-zstd: it uses native bindings, a no-no for the web.

• node-zstandard: it just spawns zstd as a child process.

• zstd-codec: it's a 1 year out of date emscripten compiled version of zstd, which weighs almost 1MB min+gzipped and can't be tree-shaken.

• zstddec: this seems the best port to webassembly, it only supports decoding and weighs ~17kb, but it has the following pretty concerning warning in its readme:

  Limitations: The decoder may fail with the error wasm function signature contains illegal type when the uncompressedSize is not known in advance and given to the decode() method. This is presumably a bug in the WASM bindings, which I am not yet sure how to fix.

Basically there currently isn't a great option for decoding zstd files on the web and I'd love to see one from fflate.

[101arrowz]

I'm well aware of the Zstandard situation on the web. I've actually already been trying to build a Zstandard encoder/decoder in JS, but unfortunately it's a massive undertaking not only due to the higher complexity of the algorithm but also because JS simply has a speed limit. The reason fflate is faster than WASM Zlib is due to the high cost of data transfer to and from WASM; however, when something is complex enough, WASM's faster execution simply wins. I'm pretty sure WASM bindings will be at worst similar and at best outright better in performance than JS.

If you are looking for decent Zstd bindings, check out these. 183kB minzipped, which isn't terrible, and there's no issue with bad compression. (By the way, the author also wrote the best Zlib WASM bindings I've seen, which are still typically slower than fflate and have a greater impact on bundle size).

Regardless, I'm still going to work on a Zstandard encoder and decoder, but it won't be part of fflate: as the name suggests, fflate is meant to be only for DEFLATE and its wrappers (ZIP, Zlib, and GZIP). I'm working on fzstd on the side and will update here when I publish the first version.

[fabiospampinato]

Awesome! Thanks for the thorough answer and good news.

[fabiospampinato]

I'm working on fzstd on the side and will update here when I publish the first version.

@101arrowz Could you guess at how long that will take? Like are we talking about days, weeks, months?

[101arrowz]

Maybe 3-6 weeks, assuming nothing comes up. Since I don't have a personal need for Zstandard, I probably won't be able to stick to an exact schedule, but that's my estimation. In the meantime I recommend the Zstandard bindings I linked; they're pretty fast and are a good compromise if Zstandard compresses that much better than GZIP.

[fr33tux]

Will be happy to help testing whenever you have something to share! 👍

[fabiospampinato]

In the meantime I've made a ~27kb decompressor-only port of zstd v1.5 to wasm that should be as fast as the one you linked (it's compiled with -O3 and it's much lighter to load actually), I'm not too sure about reliability though, but thinking about this I have the luxury of only needing to decompress some files I'm compressing myself, so I can just check if it's able to decompress those files before shipping them.

[101arrowz]

I've created a working Zstandard decompressor, but it has subpar performance (7x slower than WebAssembly). I'll try to improve performance a bit before publishing.

[101arrowz]

Unfortunately my optimization efforts haven't yielded meaningful improvements. Performance varies from 2x to 5x slower than WASM (though the bundle size could make up for this; it's 7kB minified). I'll add support for dictionaries before publishing (should be pretty soon now).

[fabiospampinato]

I'm benchmarking against the WASM bindings you compiled (zstandard-wasm) with the standard import, which I think defaults to speed, on Node.js. No idea if that uses SIMD or not.

Yeah that's optimized for speed, without SIMD instructions though.

Of course, the port supports streaming

That's great, personally I think I'll switch to your port once that's available 👍

[101arrowz]

I've published fzstd v0.0.1 on NPM. I didn't have time to complete the streaming API but the internals definitely support it in theory, and the non-streaming decompressor still supports streamed archives. I tested this on 10000 Zstandard files and it worked well for the most part, generally reaching speeds of 50-150MB/s for most archives and up to 1GB/s for archives with less FSE-compressed data. Node.js and browsers as old as IE11 still supported. Average performance is 2x slower than WASM (better than I thought I could get actually). A really interesting possible use case for this is hyper-compressed scripts on the web - load the 3 kB decompressor and a bit of boilerplate, compress your JS with Zstandard, and if that's smaller than gzip compression it could actually be really useful for some sites.

import { decompress } from 'fzstd';
const out = decompress(fs.readFileSync('whatever.zst'));
const customOutBuf = new Uint8Array(100000);
decompress(new Uint8Array(await (await fetch('/myFile.zst')).arrayBuffer()), customOutBuf);

[fabiospampinato]

Awesome! Thank you for your work on this! I'm personally going to be waiting for streaming support, and then most probably switch to this from the WASM port for better memory characteristics and smaller bundles, zstd is already so fast for my use case.

A really interesting possible use case for this is hyper-compressed scripts on the web

I was thinking the same the other day, this could be applied to stuff like Electron apps too, it will shave tens of megabytes from bundles for some apps and IMO potentially improve startup performance too (~50~75% less stuff to read and real-time decompression), but the maintainers there thought differently, maybe this doesn't actually work in practice 🤷‍♂️ I should take some measurements. Ref: electron/electron#29556

[101arrowz]

I think the distinction is disk vs web. On the web compression is crucial because the network is slow and bandwidth is expensive; however on disk usually that's not usually the case. Something like UPX can be applied to the final Electron binary without direct support in Electron (though I agree it would be nice for QoL).

[101arrowz]

Finished the streaming decompressor @fabiospampinato and anyone else who needs it; it's available as of v0.0.3 on NPM. I don't expect it to be bug-free (unlike the synchronous decompressor, which I tested extensively), so be cautious and please report bugs. Docs are updated and provide this example:

import { Decompress } from 'fzstd';
let outChunks = [];
const stream = new Decompress((chunk, isLast) => {
  // Add to list of output chunks
  outChunks.push(chunk);
  // Log after all chunks decompressed
  if (isLast) {
    console.log('Output chunks:', outChunks);
  }
});

// You can also attach the data handler separately if you don't want to
// do so in the constructor.
stream.ondata = (chunk, final) => { ... }

// Since this is synchronous, all errors will be thrown by stream.push()
stream.push(chunk1);
stream.push(chunk2);
...
// Last chunk must have the second parameter true
stream.push(chunkLast, true);

// Alternatively, you can push every data chunk normally and push an empty
// chunk at the end:
// stream.push(chunkLast);
// stream.push(new Uint8Array(0), true);

[yansenlei]

How to implement ZSTD method compression, looking forward to reply. @fabiospampinato @101arrowz