[GLUTEN-7860][CORE] In shuffle writer, replace MemoryMappedFile with ReadableFile to avoid OOM #7861
Conversation
|
cc @kecookier |
zhztheplayer
changed the title
|
/Benchmark Velox |
1 similar comment
|
/Benchmark Velox |
|
/Benchmark Velox |
| @@ -73,7 +73,7 @@ void Spill::insertPayload( | |||
|
|
|||
| void Spill::openSpillFile() { | |||
| if (!is_) { | |||
| GLUTEN_ASSIGN_OR_THROW(is_, arrow::io::MemoryMappedFile::Open(spillFile_, arrow::io::FileMode::READ)); | |||
| GLUTEN_ASSIGN_OR_THROW(is_, arrow::io::ReadableFile::Open(spillFile_)); | |||
There was a problem hiding this comment.
Is the API implemented with buffered read?
Not sure whether https://github.com/apache/arrow/blob/main/cpp/src/arrow/io/buffered.h may help here.
There was a problem hiding this comment.
Spill merge needn't buffer
@zhztheplayer There's no shuffle spill on jenkins. The change won't be tested. |
Thought we always rely on Spark-controlled spill in shuffle. Does Jenkins CI always have enough memory for all shuffle data? |
This comment was marked as off-topic.
This comment was marked as off-topic.
Is it because the spill will be triggered on other operators in the pipeline? Like a sort + shuffle. Will the sort be triggered or shuffle? |
|
@zhztheplayer @marin-ma can we create a query and config to test it? |
What changes were proposed in this pull request?
This pr fixed #7860 by replacing
mmapwithreadto read spill file.How was this patch tested?