-
Notifications
You must be signed in to change notification settings - Fork 13
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
gmxapi.commandline_operation
seems to have memory leaks
#274
Comments
Here is the content of
And here is the content of
Please let me know if my interpretation of the figure is correct and if further information is needed to help troubleshoot. Thanks a lot! |
Here are some additional but less relevant interpretations of the figure:
|
Thanks! This is very interesting. This is a sort of scaling situation that has certainly not been deeply explored with gmxapi and gmxapi.commandline_operation. For the wall time, I'll probably have to reproduce and profile pretty broadly. It could be any combination of things ranging from memory (re)allocation to filesystem interactions, as well as unsophisticated order(N) logic in gmxapi. For the memory, I can say right away that there are some places in the python module that don't deallocate or reuse memory when they could. In particular, gmxapi.commandline_operation relies on several dynamically defined functions and types that don't get cached/reused and may not get deleted until the interpreter shuts down. I'll definitely want to make sure that scalems.executable does not have the same slop, and you've pointed out some things to test. Overall, I had planned that the workflow management part of both gmxapi and scalems would keep a hash in memory of all tasks and results that the script is aware of, but if adding to the mapping gets problematically slow or memory intensive as the number of workflow items exceeds 10^5 or 10^6, we may need a plan for a lighter weight bookkeeping method. Additionally, both scalems and gmxapi will need to migrate to a hierarchical approach for artifacts and task directories. HPC admins will be livid if we routinely cause directories with tens of thousands of items! |
To test the assumption that
gmxapi.commandline_operation
has memory leaks, I performed the following two tests:os.system
to run the GROMACS grompp for 20000 times to generate 20000 tpr files. (SeeTest_A.py
below.)gmxapi.commandline_operation
to run GROMACS grompp commands to generate 20000 tp files. (SeeTest_B.py
below.)Each of the 20000 iterations in each test was timed. Both the executions of
Test_A.py
andTest_B.py
were memory-profiled by themprof run
command (e.g.mprof run python Test_A.py
) enabled by memory-profiler, which measured the memory usage every 0.1 seconds. Below I plotted the wall time per tpr generation against the number of grompp commands executed (left) the memory usage as a function of time (right).It can be seen from the figure above that memory usage increased as a function of time when
gmxapi.commandline_operation
was used. On the other hand, in Test A, whereos.system
was used, the memory usage remained roughly constant, which to my understanding is because thatos.system
ran the GROMACS grompp command as a separate process and the allocated memory was released once each grompp command was finished. That is,gmxapi.commandline_operation
seems to have memory leaks.I'll paste
Test_A.py
andTest_B.py
in the next comment.The text was updated successfully, but these errors were encountered: