You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Please check that this issue hasn't been reported before.
I searched previous Bug Reports didn't find any similar reports.
Expected Behavior
Should be able to train just like using small dataset.
Current behaviour
Throw error such as
[rank0]: File "pyarrow/table.pxi", line 4387, in pyarrow.lib.Table.combine_chunks
[rank0]: File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
[rank0]: File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
[rank0]: pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays
Please check that this issue hasn't been reported before.
Expected Behavior
Should be able to train just like using small dataset.
Current behaviour
Throw error such as
Steps to reproduce
Prepare a dataset, configure it as
dataset-test.json
has more than 650K rows, and each json object is something likeDelete some items in this dataset to make it less or equal to 650K rows will fix this error.
Config yaml
Possible solution
NA
Which Operating Systems are you using?
Python Version
3.10
axolotl branch-commit
main
Acknowledgements
The text was updated successfully, but these errors were encountered: