Size limit for gzip decompression

[mvallet91]

Is there a limitation on the size of files to decompress in the browser?

I'm testing fflate for a project where we decompress and process some log files (file.log.gz) in the browser and calculate some indicators. Most files are between 50MB and 100MB gzipped, and it works really well!

The problem starts with files around 200MB deflated (~2.1GB inflated) where I get the following error:

RangeError: Array buffer allocation failed In Chrome 87 (64-bit)
RangeError: invalid array length in Firefox 84.0 (64-bit)

The error happens in the decompress step, a sample code looks like this:

let reader = new FileReader();

reader.onload = function(event) {
  let content = fflate.decompress(new Uint8Array(event.target.result), (err, dat) => {
        if (err) {
          console.log(err)
        } else {
          let stringContent = fflate.strFromU8(dat);
          
          // do things with the string
          let lines = stringContent.split('\n');
          console.log('Lines in file:', lines.length);
        }
  });  
reader.readAsArrayBuffer(file);

[101arrowz]

There's a hard limit on how big a single typed array can be. 2.1 GB is roughly the 2³¹ byte limit that exists in all browsers for all typed arrays. Remember that you're housing 2.1 GB of data in memory at once.

Your use seems perfect for streaming the data in one chunk at a time.

let reader = new FileReader();

reader.onload = function(event) {
  const buffer = new Uint8Array(event.target.result);
  let totalLines = 0;
  const strm = new fflate.AsyncDecompress((err, dat) => {
    // If this is all standard ASCII:
    const stringContent = fflate.strFromU8(dat);
    totalLines += stringContent.split('\n').length;
  });
  let i = 0;
  for (; i < buffer.length - 512000; i += 512000) {
    strm.push(buffer.slice(i, i + 512000));
  }
  strm.push(buffer.slice(i), true);
});  
reader.readAsArrayBuffer(file);

Memory usage will cap out at 512 kB with this, and the decompression should work up to 2GB compressed files. After that, you'll need to stream the data in as well.

[101arrowz]

I wanted to give an update to this with the latest version of fflate @mvallet91. There's a new DecodeUTF8 class that will prevent certain errors with special characters. All you need to do is use it instead of strFromU8. Also, you should consider file.stream if you're using new browsers.

const stream = file.stream();
const reader = stream.getReader();
const utf8Stream = new fflate.DecodeUTF8((str, final) => {
  // This is where your string data will arrive
  console.log(str, str.split('\n'));
});
const dcmpStream = new fflate.AsyncDecompress((dat, final) => {
  // Chain streams
  utf8Stream.push(dat, final);
});
reader.read().then(({value, done}) => {
  if (done) dcmpStream.push(new Uint8Array(0), true);
  else dcmpStream.push(value);
})

This is much higher performance than even your previous solution because at no point in time is more than 64kB of memory used. For context, your original solution failed because it was over 2GB, my previous solution used 200MB. Hope this helps!

[mvallet91]

Awesome! I'll try it out! I already had your previous answer working.

For my application I do have to read as many lines as possible, since I build "learner sessions" from the events in the logfile. It was originally a Python script reading the whole unzipped log and creating these learner sessions for each day (daily log file). By chunking into more manageable pieces, some sessions are interrupted, so I don't know how streaming the file will affect this.