Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(compression): add ZSTD compression support for both Node.js and the browser #78

Conversation

Mickael-van-der-Beek
Copy link

Problem

As described in the issue I opened here: #77, the amount of compression algorithms that are described in the Parquet file format specification increased and now include the ZSTD method.
This compression algorithm is currently not supported by this package.

Solution

I quickly added ZSTD support by using an NPM module both compatible with the browser and Node.js called @oneidentity/zstd-js.

This PR simply calls the compression and decompression methods of the underlying ZSTD library.

One point of interest is that the Node.js use-case requires some type conversions due to the accepted input types of the ZSTD library.

@wilwade
Copy link
Member

wilwade commented Mar 7, 2023

@Mickael-van-der-Beek Thanks for the PR and sorry for the lag!

Added some simple tests to parallel the other compression integration tests, but at least locally for me.

Thoughts?

failed to asynchronously prepare wasm: RangeError: WebAssembly.instantiate(): Out of memory: wasm memory
RangeError: WebAssembly.instantiate(): Out of memory: wasm memory

See all the errors here: https://github.com/LibertyDSNP/parquetjs/actions/runs/4355807748/jobs/7612934103

@wilwade
Copy link
Member

wilwade commented May 2, 2023

Stale

@wilwade wilwade closed this May 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants