-
Notifications
You must be signed in to change notification settings - Fork 751
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(query): window partition by spill to disk #16441
Conversation
Signed-off-by: coldWater <[email protected]>
Signed-off-by: coldWater <[email protected]>
Signed-off-by: coldWater <[email protected]>
Signed-off-by: coldWater <[email protected]>
...pelines/processors/transforms/window/partition_by/transform_window_partition_spill_writer.rs
Outdated
Show resolved
Hide resolved
Signed-off-by: coldWater <[email protected]>
Signed-off-by: coldWater <[email protected]>
Signed-off-by: coldWater <[email protected]>
Signed-off-by: coldWater <[email protected]>
Signed-off-by: coldWater <[email protected]>
Signed-off-by: coldWater <[email protected]>
Signed-off-by: coldWater <[email protected]>
3bea4c5
to
205a1de
Compare
Signed-off-by: coldWater <[email protected]>
Is there some benchmark numbers for this PR? |
@BohuTANG |
...e/src/pipelines/processors/transforms/window/partition_by/transform_window_partition_sort.rs
Outdated
Show resolved
Hide resolved
Signed-off-by: coldWater <[email protected]>
Signed-off-by: coldWater <[email protected]>
Signed-off-by: coldWater <[email protected]>
Signed-off-by: coldWater <[email protected]>
Signed-off-by: coldWater <[email protected]>
Signed-off-by: coldWater <[email protected]>
Docker Image for PR
|
Signed-off-by: coldWater <[email protected]>
9f2dd72
to
8b9aee7
Compare
Docker Image for PR
|
After testing, it was found that the local spill that occurred on the local machine and the cloud had the problem of memory not being released in time, which led to performance degradation. |
If so, this PR should revert from main branch. |
I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/
Summary
part of #15328
Changes:
How to enable:
set window_partition_spilling_to_disk_bytes_limit = 10 * 1024 * 1024 * 1024
(10G), this is the maximum disk space allowed for each request, when the conditions of spill are met, the local disk will be used in preferenceTests
Type of change
This change is