Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Document CUDF host compression option #11961

Open
revans2 opened this issue Jan 13, 2025 · 0 comments
Open

[FEA] Document CUDF host compression option #11961

revans2 opened this issue Jan 13, 2025 · 0 comments
Labels
documentation Improvements or additions to documentation feature request New feature or request

Comments

@revans2
Copy link
Collaborator

revans2 commented Jan 13, 2025

Is your feature request related to a problem? Please describe.
CUDF recently added in the option to compress gzip orc and parquet files on the CPU instead of the GPU using a thread pool.

rapidsai/cudf#17656

This is controlled by two environment variables.

  • LIBCUDF_HOST_COMPRESSION_NUM_THREADS to set the number of threads that the pool will use (this is a static pool that appears to be shared by everyone).
  • LIBCUDF_USE_HOST_COMPRESSION which controls is we should try and use host compression if we have a choice.

Right now only gzip is available, but that might change in the future. At a minimum we should have this as an option to use to debug if nvcomp is having issues or not.

@revans2 revans2 added ? - Needs Triage Need team to review and classify feature request New feature or request labels Jan 13, 2025
@mattahrens mattahrens added documentation Improvements or additions to documentation and removed ? - Needs Triage Need team to review and classify labels Jan 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation feature request New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants