Proper way of tearing down cuRobo (e.g. freeing all GPU memory)

[ChengshuLi]

I was hoping to create a MotionGen object, do some motion planning, remove it (freeing all GPU memory), and then create a new MotionGen object later.

Currently, the GPU memory isn't being freed up properly. Any suggestions? Many thanks!!!

mg = curobo.wrap.reacher.motion_gen.MotionGen(...)
result = mg.plan_batch(...)
del mg
# GPU memory isn't freed properly. Something needs to be done here...
new_mg = curobo.wrap.reacher.motion_gen.MotionGen(...)

[balakumar-s]

Does this help? https://pytorch.org/docs/stable/generated/torch.cuda.empty_cache.html

[ChengshuLi]

This doesn't help.

I also tried a few things:

1. setting use_cuda_graph=False
2. get rid of torch.jit by always returning empty_decorator in get_torch_jit_decorator
3. calling gc.collect()

None of these work actually. Any advice will be greatly appreciated!!

[balakumar-s]

cuRobo doesn't manage it's CUDA memory, it relies on pyTorch's memory manager to do it.

[ChengshuLi]

@balakumar-s I figured out the reason. curobo itself works fine.
The culprit is a setting in Isaac Sim called "/app/extensions/pathStatCacheEnabled", which caches all importlib's os.stat call.

If I start Isaac Sim first, and then create a curobo MotionGen instance, the MotionGen instance does a few dynamic import (e.g. MultivariateNormal internally calls from torch.fx.experimental.symbolic_shapes import guard_size_oblivious). Isaac Sim will cache this import. Now if I del motion_gen_instance, the memory will NOT be freed because Isaac Sim still keep a reference to everything in the MotionGen instance. It's a crazy bug :(