Infinite RAM usage can exhaust system

[paulmillr]

RAM usage seems to be unlimited with the module. RAM requirements need to be clarified somewhere, because crashing apps is not ok.

[thecodrr]

Fdir just crawls the directories, and yes, if you have an insane amount of items it may end up OOM-ing your app. What's the suggested outcome here?

[paulmillr]

Readdirp have limits on ram usage due to its architecture. It can’t exceed X.

[43081j]

readdirp will just emit files/dirs, right? so it won't really use much memory, as it isn't building up a list of found paths

if the user turns the result into an array, they will hit the same issue fdir has.

you can set maxFiles possibly to limit the array size, so you won't hit any OOM issues

alternatively, maybe we could introduce an option to pass a callback which will be called each time we visit a path, and we skip building up the internal set of paths (which means you can't use group etc)

[paulmillr]

Well, stream APIs exist for a reason. Not always users want an array of 400K files. etc.