The encoded data was not valid for encoding utf-8

[denniske]

How to reproduce

https://aoe4world.com/dumps
File: Games - RM 1v1 - Season 3 - 44 MB

Grab the file url and put it into url variable.

const dump = await (await fetch(url)).arrayBuffer();
const compressed = new Uint8Array(dump);
const decompressed = decompressSync(compressed);
const origText = strFromU8(decompressed);

The problem

The following error occurs in Line 4 in strFromU8(decompressed):

TypeError: The encoded data was not valid for encoding utf-8
    at TextDecoder.decode (node:internal/encoding:448:14)
    at strFromU8 (/Users/dennis/Projects/poc_collector/node_modules/fflate/lib/node.cjs:1780:19)
    at HistoricalTask.<anonymous> (/Users/dennis/Projects/poc_collector/dist/collector/webpack:/src/task/historical.task.ts:79:39)
    at Generator.next (<anonymous>)
    at fulfilled (/Users/dennis/Projects/poc_collector/node_modules/tslib/tslib.js:166:62)
    at processTicksAndRejections (node:internal/process/task_queues:95:5) {
  code: 'ERR_ENCODING_INVALID_ENCODED_DATA'
}

I can download the file on mac os and extract it by double click in the finder and then open in visual studio code without problems. Visual studio code shows UTF-8 in the status bar.

[101arrowz]

That file is 533MB decompressed, but strings in JavaScript can be at most 512MB. In other words, it's not possible to decompress this data to a single string in JavaScript at all, no matter what library you use.

You can try to solve this by decompressing the data in a stream and using a streaming JSON parser; let me know if you want more info on how to do that.

[denniske]

Ah I see. Yes I am interested in the streaming json parser.

[101arrowz]

You may want to install this Compressions Streams polyfill with npm i compression-streams-polyfill - it's based on fflate and mimicks the Compression Streams API.

You'll also need a package for streaming JSON parsing. I'll suggest this because it works with native stream APIs.

import 'compression-streams-polyfill';
import { JSONParser } from '@streamparser/json-whatwg';

const byteCounter = new TransformStream({
  transform(dat, ctrl) {
    len += dat.byteLength;
    console.log('Bytes downloaded: ', len);
    ctrl.enqueue(dat);
  }
});

let len = 0;
const { body } = await fetch(url);
const result = body
  // Can remove byteCounter if you want
  .pipeThrough(byteCounter)
  .pipeThrough(new DecompressionStream('gzip'))
  .pipeThrough(new JSONParser({ stringBufferSize: undefined, paths: ['$.*'], keepStack: false }));

let values = [];
const reader = result.getReader();
while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  // Use the value variable as you wish - this is an entry in the file
}

You can't push all the values into one big array by the way - that will cause the engine to run out of memory.

[denniske]

Thanks for the code snippet. It works great for big files and it is really clean code compared to what I found with google 😅
There is one issue about the performance of the json parser when streaming:

When parsing one of the smaller files (Games - QM 1v1 - 2023 Q1) at once with fflate it takes 3 seconds. But with the streaming it takes multiple minutes.

I have put the byteCounter in the first position as well as after the DecompressionStreamand I see that it gets downloaded and compressed within seconds. But when logging the output with:

const { done, value } = await reader.read();
if (done) break;

this.lineNumber++;
if (this.lineNumber % 10000 == 0) console.log(this.lineNumber);

I can see that parsing the json takes a huge amount of time. Are there performance enhancements I can use for the JSONParser?

[101arrowz]

I don't know of any high-performance JSON parsing solutions that work with streams, but since your data fortunately seems to be newline-delimited, you can simply parse individual JSON objects at each newline. In fact with this solution it seems to even be loadable into an array in memory. You may need to polyfill TextDecoderStream depending on your target environment.

import 'compression-streams-polyfill';

const { body } = await fetch(url);

let strbuf = '';
const result = body
  .pipeThrough(new DecompressionStream('gzip'))
  // polyfills for this are easy to find if needed
  .pipeThrough(new TextDecoderStream())
  .pipeThrough(new TransformStream({
    transform(str, ctrl) {
      const priorLen = strbuf.length;
      strbuf += str;

      let lastIndex = 0;
      for (
        let i = strbuf.indexOf('\n', priorLen);
        i !== -1;
        lastIndex = i + 1, i = strbuf.indexOf('\n', lastIndex)
      ) {
        if (strbuf[lastIndex] === '[' || strbuf[lastIndex] === ']') {
          // start/end line
          continue;
        }

        // slice to newline, minus 1 character if there's a comma
        const part = strbuf.slice(lastIndex, i - +(strbuf[i - 1] === ','));
        ctrl.enqueue(JSON.parse(part));
      }

      strbuf = strbuf.slice(lastIndex);
    },
    flush() {
      if (strbuf !== '') throw new TypeError('failed to parse');
    }
  }));

const values = [];
const reader = result.getReader();
while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  values.push(value);
}

console.log(values); // all in memory

[denniske]

Thank you! That works with all the files and is very fast 👍