You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It seems that read_tsv cannot read files larger than 4GB if they are gunzipped. I did some calculations, and it seemed like it got to about 4GB of unpacked data and then ended reading, meaning that the last 150k lines were not read. I suspect there is an unsigned 32-bit integer somewhere in the code, but I cannot be sure ofcourse. This was on Windows. I think the problem exists for any huge file that is gunzipped. It didn't crash or anything, just gave too few lines. If I unzipped the file, I could read all of it without a problem. I don't think I ran out of memory either. Hard to provide an example, I cannot share the file, but I would guess any file large enough will work.
The text was updated successfully, but these errors were encountered:
Hi,
It seems that read_tsv cannot read files larger than 4GB if they are gunzipped. I did some calculations, and it seemed like it got to about 4GB of unpacked data and then ended reading, meaning that the last 150k lines were not read. I suspect there is an unsigned 32-bit integer somewhere in the code, but I cannot be sure ofcourse. This was on Windows. I think the problem exists for any huge file that is gunzipped. It didn't crash or anything, just gave too few lines. If I unzipped the file, I could read all of it without a problem. I don't think I ran out of memory either. Hard to provide an example, I cannot share the file, but I would guess any file large enough will work.
The text was updated successfully, but these errors were encountered: