You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
CUDF recently added in the option to compress gzip orc and parquet files on the CPU instead of the GPU using a thread pool.
LIBCUDF_HOST_COMPRESSION_NUM_THREADS to set the number of threads that the pool will use (this is a static pool that appears to be shared by everyone).
LIBCUDF_USE_HOST_COMPRESSION which controls is we should try and use host compression if we have a choice.
Right now only gzip is available, but that might change in the future. At a minimum we should have this as an option to use to debug if nvcomp is having issues or not.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
CUDF recently added in the option to compress gzip orc and parquet files on the CPU instead of the GPU using a thread pool.
rapidsai/cudf#17656
This is controlled by two environment variables.
LIBCUDF_HOST_COMPRESSION_NUM_THREADS
to set the number of threads that the pool will use (this is a static pool that appears to be shared by everyone).LIBCUDF_USE_HOST_COMPRESSION
which controls is we should try and use host compression if we have a choice.Right now only gzip is available, but that might change in the future. At a minimum we should have this as an option to use to debug if nvcomp is having issues or not.
The text was updated successfully, but these errors were encountered: