-
Notifications
You must be signed in to change notification settings - Fork 435
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[GLUTEN-7750][VL]: store unsafe batches data #7902
base: main
Are you sure you want to change the base?
Conversation
Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues? https://github.com/apache/incubator-gluten/issues Then could you also rename commit message and pull request title in the following format?
See also: |
val recordLength = 2L * uaoSize + inputLength + 8L | ||
|
||
UnsafeAlignedOffset.putSize(base, offset, inputLength + uaoSize) | ||
offset += 2L * uaoSize |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure if I'm calculating the offset
correctly
@Zand100 Is there any test case can be added for this change? |
val columnVector = new OffHeapColumnVector(batch.numElements(), DataTypes.BinaryType) | ||
columnVector.putByteArray(batchId, batch.toByteArray, batch.getBaseOffset.toInt, batch.numElements) | ||
val columnarBatch = new ColumnarBatch(Array(columnVector)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@zhztheplayer Is it alright to create the ColumnarBatch
this way, using OffHeapColumnVector
and constructing a new ColumnarBatch
directly from that?
I'm planning to write unit and integration tests. I'm still getting familiar with the code. I'm guessing for |
What changes were proposed in this pull request?
Adds a binary container implementing a Spark
MemoryConsumer
to be used instead ofArray[Array[Bytes]]
.(Fixes: #7750)
How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)