Heap memory usage lower than resident size. How to track the leaks ?

[ashajkofci]

First, thanks to the team who developed Memray ! It's a formidable tool.

Then, I am profiling a Pyside2 project (with QThreads) that leaks memory. The issue here is that the heap memory does not leak, but the allocator does not seem to want to give the memory to the OS.

Memray cannot see where that memory went since it only see a 1.1GB peak usage, which is normal. In reality, this process used more than 2GB and continues to use memory as it is used.
. Do you have any idea on how I could debug this issue ?

Many thanks

[pablogsal]

. Do you have any idea on how I could debug this issue ?

Debugging "leaking" resident size programs is always a bit of a challenge because you are dealing with underlying allocators most of the time and unless you have any visibility into those is very difficult to diagnose. You have some possibilities if you are using glibc you can use the malloc_stats call to get some statistics on the internal allocator to see if fragmentation may be a problem. Check this for more information.

If you are feeling adventurous, glibc has a bunch of tunables that you can try to help with fragmentation if that turns out to be the case:

https://www.gnu.org/software/libc/manual/html_node/Malloc-Tunable-Parameters.html

Another possibility is that memray cannot see the source of the allocations because you application is using APIs that we are not tracking. To know if that's the case, you can try to share with us some minimal reproducer so we can investigate in our side, although I don't think that is very likely to be the case.

[jstriebel]

One thing that comes to my mind is shared memory, which counts towards the RSS, but is probably not tracked by memray directly.

[pablogsal]

One thing that comes to my mind is shared memory, which counts towards the RSS, but is probably not tracked by memray directly.

Indeed, is not tracked by memray

[ashajkofci]

Hi ! Thanks for your answers ! After a while I managed to fix half of the leaks with a Py_DECREF(points); at the end of a C++ extension (see stardist/stardist#223 btw I don't know why the author couldn't compile my code because it just does :D).

Is C++ allocated memory a shared memory ? I think there are ways to give ownership of the memory to only C++ and only python but that does not change a lot.

Additionnal question: do you think I should migrate this academic spaghetti code to pybind11 ? does it allocate memory better than Python.h ?