Using memray to determine the existence of memray leakage

[EricLin89]

memray flamegraph --leak is useful when you know there's a memory leak problem and you want to find out the leaking code.
In my use case, I want to use memray to see whether there is a memory leak.
As we know, python memory allocator is more likely to hold the memory even after objects are destroyed, so if we only look at the memory usage of a process from outside, we always see a curve that grows slowly by time, which makes us difficult to determine whether there's actually a memory leak.
With memray we can see the behavior of a process from an internal view, which does a great help. However, I still see some limitations:

1. When profiling a process that I'm pretty sure there's no memory leak, I still see the heap size slowly grows by time, from the graph produced by memray flamegraph --leak, even after I set PYTHONMALLOC to malloc. I'm expecting to see heap size stable on this curcumstance but in fact it's not the case. I'm not sure if it's related to garbage collection, maybe memray can take garbage collection into consideration?

2. When hunting down memory leak, I think it's more useful to produce memory allocation diffs between timestamps. In this way we can see clearly who is leaking memory. Especially when the leaked memory is relatively small compared to the "base" memory usage of a process.

[godlygeek]

1. When profiling a process that I'm pretty sure there's no memory leak, I still see the heap size slowly grows by time, from the graph produced by memray flamegraph --leak, even after I set PYTHONMALLOC to malloc. I'm expecting to see heap size stable on this curcumstance

If the reported heap size is growing, it does tell you that Python is allocating memory from the operating system and not returning it. That's not necessarily wrong, though.

I'm not sure if it's related to garbage collection, maybe memray can take garbage collection into consideration?

It seems unlikely to be related to garbage collection to me. Heap fragmentation, for instance, can cause a program to hold on to relatively large amounts of memory even when it only needs a small bit of what's on each page. If the reported heap memory is continuously growing, though, it tells me that the process is continuously requesting more memory from the OS and not freeing all of it.

2. When hunting down memory leak, I think it's more useful to produce memory allocation diffs between timestamps. In this way we can see clearly who is leaking memory. Especially when the leaked memory is relatively small compared to the "base" memory usage of a process.

That's exactly the feature that temporal flame graphs provide. If you generate a flame graph with memray flamegraph --leaks --temporal, you'll be able to drag some sliders around to analyze what allocations were made within any given window of time (down to about 10ms granularity) and not freed by the end of that window. Make sure that your capture file was generated with either PYTHONMALLOC=malloc set or --trace-python-allocators whenever you generate a --leaks flame graph.

[EricLin89]

Heap fragmentation, for instance, can cause a program to hold on to relatively large amounts of memory even when it only needs a small bit of what's on each page

Is "heap fragmentation" you mentioned here caused by pymalloc, then I can eliminate it by setting PYTHONMALLOC=malloc or --trace-python-allocators?
In my understanding, the "heap size" curve in flamegraph report should correspond to the size of memory the profiled code asks from python interpreter. It should have nothing to do with how much memory the OS actually allocates for it. Is it correct?

[godlygeek]

In my understanding, the "heap size" curve in flamegraph report should correspond to the size of memory the profiled code asks from python interpreter. It should have nothing to do with how much memory the OS actually allocates for it. Is it correct?

That's correct. Thinking more about this, nevermind what I said about heap fragmentation. That's a factor that would account for an ever-growing RSS line on the graph, but not an ever-growing heap line on the graph.

The heap size curve is just tracking the number of bytes allocated through the allocators we interpose and not yet freed. If you're using --trace-python-allocators, we interpose the pymalloc allocator in addition to malloc and friends, so when you set that flag, we'll see the (more frequent) calls to allocate individual objects with the pymalloc allocator, instead of just the (less frequent) calls that the pymalloc allocator itself makes to malloc in order to grow its memory pool. If you use PYTHONMALLOC=malloc, you disable the pymalloc allocator entirely, so there is no pool, and we'll see each allocation of a Python object as a direct call to malloc by the interpreter. The end result of either these options is nearly the same - Memray will have visibility into each allocation of a Python object, albeit for different reasons.

If you generate a capture file with PYTHONMALLOC=malloc memray run yourscript.py and then generate a temporal flame graph from it using memray flamegraph --leaks --temporal, it will be possible to see what objects are allocated within any given time range (down to about 10ms granularity) and not freed by the end of that time range.