Unzip and parse downloaded big csv file

[reju1021]

Hi,

I spent lots of time trying to figure this out on my own, but I did not succeed:

I'm trying to download a big zipped csv file that I want to unzip and parse. The zipped file is just small enough to fit into main memory, however the unzipped file leads to a heap out of memory. It's a node.js backend so writing to disk is no problem. I figure streaming and or writing to / reading from disk is the solution.

Can I download the zipped file in chunks, unzip them on arrival, translate them to strings and write it to disk? I figure writing to disk has to be done because since it's a csv I can only parse full rows. So directly parsing the chunks won't work when a row is divided between two chunks. Is is better to write the string to disk or directly the unzipped UInt8Array? Is the order guaranteed? I.e. can I be assured that the header row remains the first row in the unzipped csv file when it's split into chunks?

Is there any code snippet which solves a similar problem, I can have a look at?

Is there a better approach?

Thanks a lot! :)

[reju1021]

Came eventually up with my own solution:

    const res = await fetch(url);
    const writeStream = fs.createWriteStream(fileNameToBeWritten);
    const nodeStream = Readable.fromWeb(res.body as unknown as ReadableStream);
    await finished(nodeStream.pipe(writeStream));

   await new Promise<void>((resolve, reject) => {
      const unzipper = new Unzip();

      unzipper.register(UnzipInflate);

      unzipper.onfile = (file) => {

        file.ondata = (err, chunk, final) => {
          if (err) {
            this.logger.error(err);
            file.terminate();
            reject(err);
          }

          const str = strFromU8(chunk);

          fs.appendFileSync(fileNameToBeWritten, str);

          if (final) {
            resolve();
          }
        };
        file.start();
      };

      const readStream = fs.createReadStream(fileNameToBeRead);
      readStream.on('data', (chunk) => {
        unzipper.push(chunk as Buffer);
      });
      readStream.on('end', () => {
        unzipper.push(new Uint8Array(), true);
      });
      readStream.on('error', (err) => {
        this.logger.error(err);
        throw err;
      });
    });