force garbage collection after large objects are removed

[cjyetman]

By forcing garbage collection after large objects are removed, significantly better memory management is achieved.

I don't relish doing this, but it works...

Before

After

[jdhoffa]

NIT: Maybe we should...
Re-write data prep in a non-gc'd language where we can explicitly and safely manage memory, and as a bonus fosters safe multi-threading in an intuitive way 😳

🦀 🦀 🦀 🦀
crab walks out of the room

[jdhoffa]

Approving as a "fine if we HAVE to" approval. And on the condition that we open a new issue to further explore a better way to get memory use within a reasonable limit.

There has to be a better way! But agree that this is an effective band-aid for now.

[cjyetman]

NIT: Maybe we should... Re-write data prep in a non-gc'd language where we can explicitly and safely manage memory, and as a bonus fosters safe multi-threading in an intuitive way 😳

@jdhoffa do you know how to do that?

[jdhoffa]

NIT: Maybe we should... Re-write data prep in a non-gc'd language where we can explicitly and safely manage memory, and as a bonus fosters safe multi-threading in an intuitive way 😳

@jdhoffa do you know how to do that?

Today, no certainly not. But I'd be very happy to give it a shot as a {pedagogical, experimental, pet project} type prototype if we think it's valuable.

[cjyetman]

just a note: using gc() is not total witchcraft... it's just explicitly determining when garbage collection occurs rather than waiting for R to guess when it's appropriate, which is a perfectly legitimate thing to do when you're using a lot of memory and you know for sure when is an appropriate time to run garbage collection

in fact, in some (many?) languages, you have to decide if/when to do it

[jdhoffa]

I understand. I think the reliance on a garbage collector (either intentionally invoked or passively relied on) is gonna result in a memory hit. But I'm not even sure that's the biggest problem here frankly.

I'd love to explore #94 more and see if we can get better performance from that, I imagine we can

[AlexAxthelm]

DNLGTM (Does not look good to me)

Approved despite my visceral reaction.

[cjyetman]

I'm happy to not merge this also. It's just an easy option for significantly improving the memory footprint, which has been a big problem for @jdhoffa and @AlexAxthelm running it.

[jdhoffa]

Oh what? I very much do approve it! I just wanted to make sure we documented the follow-up steps (which was done in #141).

There's nothing functionally wrong with this, only NIT/ NB/style/"visceral" hesitancy

But for the record, I would much rather this be merged then the status quo of not being able to run data prep

[AlexAxthelm]

But for the record, I would much rather this be merged then the status quo of not being able to run data prep

Same. My approval stands.

[cjyetman]

ok, sorry... I misunderstood the hesitancy... re-opened and will merge

[AlexAxthelm]

😆 Just because I don't like it doesn't mean it's wrong.