Is there any profiling tools on memory usage?

[Luke-Skycrawler]

I keep getting

Warp CUDA error 700: an illegal memory access was encountered (D:\repos\warp\warp\native\warp.cu:245)

when I try to crank up the resolution of my warp simulation. I checked out-of-bound access and excluded that possibility, and I'm thinking it's because my gpu is running out of memory. Is there any profiling tools that can show how much memory is allocated, or show a message whenever some cuda array is allocated or released? Any help is appreciated.

I'm quite concerned about possible memory leakage, as I have not been discrete in memory allocation. For example, I allocated some memory in my constructor

self.arr = wp.zeros(...)

and in another method I just overwrite it with

self.arr = wp.from_numpy(SOME_NUMPY_ARRAY)

I just don't know if that causes any duplicate allocation under the carpet. Again, the compiler doesn't complain, and the code works in small scale. Without the any hints it's really hard to know where to start.

[nvlukasz]

We don't currently have a way to check memory usage in Warp, but I have something in progress for that. In the meantime, you could use a tool like nvidia-smi that reports GPU usage. If running out of memory, you should see an exception from Warp that the memory allocation failed.

There are a few things you can try debugging the illegal access.

• Set wp.config.mode = "debug" at the start of your program. This will add some internal checks that may catch memory access errors.
• Also try wp.config.verify_cuda = True to add some additional checks when launching kernels.
• You could also try running your code on the "cpu" device to help narrow it down.

When you create an array like this:

self.arr = wp.zeros(...)

and later re-assign it like this:

self.arr = wp.from_numpy(SOME_NUMPY_ARRAY)

then a new array will be allocated, but the old one should get deallocated right away if there are no extra references to it. Warp uses standard Python garbage collection semantics for arrays.

Try to narrow down which Warp call generates the CUDA error and dig deeper there. If it's a kernel launch, you can try simple print() and printf() statements in your kernel to verify the data, etc.

If you're still having trouble, then it would be helpful to share some snippets of your code so we have a better understanding of what to look at next.