Can memray track subprocesses started by spawn()?

[scse-l]

When I try to track subprocesses created by spawn(), I got an empty .bin file.

import memray
import torch.multiprocessing as mp
import os

def worker(i: int):
    t = list(range(10000))
    b = list(map(lambda x: x+2, t))
    print("From worker {}".format(i))


def f():
  mp.start_processes(worker, args=(), nprocs=4, join=True, start_method='spawn')

if __name__ == '__main__':
  with memray.Tracker("output_file.bin.{}".format(os.getpid()), follow_fork=True):
    f()

Run this piece of code will produce 6 .bin files, most of them are empty.
The output of cmd memray tree <one of .bin file> is :

Allocation metadata
-------------------
Command line arguments: 'test.py'
Peak memory size: 0.000B
Number of allocations: 0

Biggest 10 allocations:
-----------------------
<No allocations>

If we change use fork

def f():
  mp.start_processes(worker, args=(), nprocs=4, join=True, start_method='fork')

The .bin files will not be empty:

Allocation metadata
-------------------
Command line arguments: 'test.py'
Peak memory size: 942.430KB
Number of allocations: 113

Biggest 10 allocations:
-----------------------
📂 942.305KB (100.00 %) <ROOT>  
└── [[5 frames hidden in 4 file(s)]]
    └── 📂 942.305KB (100.00 %) __init__  multiprocessing/popen_fork.py:20
        ├── 📂 942.117KB (99.98 %) _launch  multiprocessing/popen_fork.py:74
        │   ├── [[2 frames hidden in 1 file(s)]]
        │   │   └── 📂 937.039KB (99.44 %) _wrap  torch/multiprocessing/spawn.py:69
        │   │       ├── [[1 frames hidden in 1 file(s)]]
        │   │       │   └── 📄 512.000KB (54.33 %) <lambda>  test.py:8
        │   │       └── 📄 343.938KB (36.50 %) worker  test.py:7
        │   ├── [[1 frames hidden in 1 file(s)]]
        │   │   └── 📄 4.031KB (0.43 %) _close_stdin  /multiprocessing/util.py:399
        │   └── 📄 536.000B (0.06 %) _bootstrap  /multiprocessing/process.py:290
        └── 📂 192.000B (0.02 %) _launch  /multiprocessing/popen_fork.py:70
            ├── 📄 32.000B (0.00 %) _after_fork  /python3.7/threading.py:1347
            ├── [[2 frames hidden in 1 file(s)]]
            │   └── 📄 32.000B (0.00 %) _reset_internal_locks /python3.7/threading.py:505
            └── [[2 frames hidden in 1 file(s)]]
                └── 📄 32.000B (0.00 %) RLock  /python3.7/threading.py:86

[godlygeek]

The spawn method forks a process off, and then executes the interpreter all over again in that forked process. The files get written because we successfully trace through the fork call, but they're nearly empty because soon after the fork the process image is replaced by an exec call.

So, you're definitely right that we can't trace into spawned worker processes like we do into forked ones.

[scse-l]

Got it, thanks!