Fails while running on OSX

[dnkennedy]

Hi, intrepid user here. Running through the 'walkthrough'; in the 'containers-run' step I get the following error:

(base) kennedyd@MCSMLS065401 ds000003-qc % datalad containers-run
--container-name bids-mriqc
--input sourcedata
--output .
'{inputs}' '{outputs}' participant --participant-label sub-02 -w workdir
[INFO ] Making sure inputs are available (this may take some time)
get(ok): containers/images/bids/bids-mriqc--0.16.0.sing (file) [from origin...]
[INFO ] == Command start (output follows) =====
240523-20:51:23,365 cli IMPORTANT:

Running MRIQC version 0.16.0:
  * BIDS dataset path: /Users/kennedyd/my-experiments/ds000003-qc/sourcedata.
  * Output folder: /Users/kennedyd/my-experiments/ds000003-qc.
  * Analysis levels: ['participant'].

You are using MRIQC v0.16.0, and a newer version is available: 24.0.0.
Traceback (most recent call last):
File "/usr/local/miniconda/bin/mriqc", line 10, in
sys.exit(main())
File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/cli/run.py", line 37, in main
with Manager() as mgr:
File "/usr/local/miniconda/lib/python3.7/multiprocessing/context.py", line 56, in Manager
m.start()
File "/usr/local/miniconda/lib/python3.7/multiprocessing/managers.py", line 543, in start
self._process.start()
File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 112, in start
self._popen = self._Popen(self)
File "/usr/local/miniconda/lib/python3.7/multiprocessing/context.py", line 291, in _Popen
return Popen(process_obj)
File "/usr/local/miniconda/lib/python3.7/multiprocessing/popen_forkserver.py", line 35, in init
super().init(process_obj)
File "/usr/local/miniconda/lib/python3.7/multiprocessing/popen_fork.py", line 20, in init
self._launch(process_obj)
File "/usr/local/miniconda/lib/python3.7/multiprocessing/popen_forkserver.py", line 51, in _launch
self.sentinel, w = forkserver.connect_to_new_process(self._fds)
File "/usr/local/miniconda/lib/python3.7/multiprocessing/forkserver.py", line 64, in connect_to_new_process
self.ensure_running()
File "/usr/local/miniconda/lib/python3.7/multiprocessing/forkserver.py", line 118, in ensure_running
listener.bind(address)
OSError: [Errno 22] Invalid argument
[INFO ] == Command exit (modification check follows) =====
run(error): /Users/kennedyd/my-experiments/ds000003-qc (dataset) [./containers/scripts/singularity_cmd run...]
action summary:
get (notneeded: 2, ok: 1)
run (error: 1)
save (notneeded: 3)

I'm running on a Mac: 14.1 (23B74)
Docker: 4.28.0 (139021)
Engine: Engine: 25.0.3

[yarikoptic]

FWIW, running the "monolythic" version of walkthrough fails but not like above:

(datalad) yoh@datalads-imac2 containers % bash <(sed -n -e '/^ *#!/,/^```$/p' README.md | grep -v '```')
...
> containers/scripts/freeze_versions --save-dataset=. bids-mriqc=0.16.0
I: We will be copying/freezing versions in .
I: bids-mriqc -> 0.16.0
add(ok): .datalad/config (file)
save(ok): . (dataset)
action summary:
  add (ok: 1)
  save (ok: 1)
> datalad install -d . -s https://github.com/ReproNim/ds000003-demo sourcedata
[INFO   ] Remote origin not usable by git-annex; setting annex-ignore
[INFO   ] https://github.com/ReproNim/ds000003-demo/config download failed: Not Found
[INFO   ] access to 1 dataset sibling s3-PRIVATE not auto-enabled, enable with:
|               datalad siblings -d "/private/var/folders/tk/5_lnk94x727gwq_x4z6mf1x00000gp/T/repro-8UTZo4N/ds000003-qc/sourcedata" enable -s s3-PRIVATE
install(ok): sourcedata (dataset)
add(ok): sourcedata (dataset)
add(ok): .gitmodules (file)
save(ok): . (dataset)
add(ok): .gitmodules (file)
save(ok): . (dataset)
action summary:
  add (ok: 3)
  install (ok: 1)
  save (ok: 2)
> echo workdir/
> datalad save -m 'Ignore workdir' .gitignore
add(ok): .gitignore (file)
save(ok): . (dataset)
action summary:
  add (ok: 1)
  save (ok: 1)
> datalad containers-run -n bids-mriqc --input sourcedata --output . '{inputs}' '{outputs}' participant group -w workdir
[INFO   ] Making sure inputs are available (this may take some time)
get(ok): sourcedata/sub-02/anat/sub-02_T1w.nii.gz (file) [from s3-PUBLIC...]
get(ok): sourcedata/sub-02/anat/sub-02_inplaneT2.nii.gz (file) [from s3-PUBLIC...]
get(ok): sourcedata/sub-02/func/sub-02_task-rhymejudgment_bold.nii.gz (file) [from s3-PUBLIC...]
get(ok): sourcedata/sub-13/anat/sub-13_T1w.nii.gz (file) [from s3-PUBLIC...]
get(ok): sourcedata/sub-13/anat/sub-13_inplaneT2.nii.gz (file) [from s3-PUBLIC...]
get(ok): sourcedata/sub-13/func/sub-13_task-rhymejudgment_bold.nii.gz (file) [from s3-PUBLIC...]
get(ok): containers/images/bids/bids-mriqc--0.16.0.sing (file) [from origin...]
[INFO   ] == Command start (output follows) =====
240529-19:51:23,714 cli IMPORTANT:

    Running MRIQC version 0.16.0:
      * BIDS dataset path: /var/folders/tk/5_lnk94x727gwq_x4z6mf1x00000gp/T/repro-8UTZo4N/ds000003-qc/sourcedata.
      * Output folder: /var/folders/tk/5_lnk94x727gwq_x4z6mf1x00000gp/T/repro-8UTZo4N/ds000003-qc.
      * Analysis levels: ['group', 'participant'].

You are using MRIQC v0.16.0, and a newer version is available: 24.0.0.
240529-19:51:24,443 nipype.utils WARNING:
         A newer version (1.8.4) of nipy/nipype is available. You are using 1.6.0
240529-19:51:26,343 nipype.utils WARNING:
         A newer version (1.8.4) of nipy/nipype is available. You are using 1.6.0
240529-19:51:37,408 cli WARNING:
         IMPORTANT: Anonymized quality metrics (IQMs) will be submitted to MRIQC's metrics repository. Submission of IQMs can be disabled using the ``--no-sub`` argument. Please visit https://mriqc.readthedocs.io/en/latest/dsa.html to revise MRIQC's Data Sharing Agreement.
240529-19:51:37,910 nipype.workflow WARNING:
         Some nodes exceed the total amount of memory available (1.75GB).
240529-19:51:39,117 nipype.utils WARNING:
         A newer version (1.8.4) of nipy/nipype is available. You are using 1.6.0
240529-19:51:39,120 nipype.utils WARNING:
         A newer version (1.8.4) of nipy/nipype is available. You are using 1.6.0
240529-19:51:39,135 nipype.utils WARNING:
         A newer version (1.8.4) of nipy/nipype is available. You are using 1.6.0
240529-19:51:39,225 nipype.utils WARNING:
         A newer version (1.8.4) of nipy/nipype is available. You are using 1.6.0

240529-20:00:03,917 cli WARNING:
         Building bold report: no exclude index was found
240529-20:03:12,53 cli WARNING:
         Building bold report: no exclude index was found
The use of ``template_resolution`` is deprecated
The use of ``template_resolution`` is deprecated
The use of ``template_resolution`` is deprecated
240529-20:07:23,275 nipype.workflow WARNING:
         Storing result file without outputs
240529-20:07:23,288 nipype.workflow WARNING:
         [Node] Error on "mriqc_wf.anatMRIQC.AirMaskWorkflow.invert_xfm" (/var/folders/tk/5_lnk94x727gwq_x4z6mf1x00000gp/T/repro-8UTZo4N/ds000003-qc/workdir/mriqc_wf/anatMRIQC/AirMaskWorkflow/_in_file_..var..folders..tk..5_lnk94x727gwq_x4z6mf1x00000gp..T..repro-8UTZo4N..ds000003-qc..sourcedata..sub-13..anat..sub-13_T1w.nii.gz/invert_xfm)
240529-20:07:24,497 nipype.workflow ERROR:
         Node invert_xfm.a1 failed to run on host 76f636198895.
240529-20:07:24,687 nipype.workflow ERROR:
         Saving crash info to /var/folders/tk/5_lnk94x727gwq_x4z6mf1x00000gp/T/repro-8UTZo4N/ds000003-qc/logs/crash-20240529-200724-host-user-invert_xfm.a1-9dd79aa3-82a2-4624-a99c-aeca769eed9b.txt
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 434, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 829, in _run_interface
    self.raise_exception(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 760, in raise_exception
    ).format(**runtime.dictcopy())
RuntimeError: Command:
antsApplyTransforms --default-value 0 --dimensionality 3 --float 1 --input /home/bidsapp/.cache/templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-01_desc-head_mask.nii.gz --interpolation MultiLabel --output tpl-MNI152NLin2009cAsym_res-01_desc-head_mask_trans.nii.gz --reference-image /var/folders/tk/5_lnk94x727gwq_x4z6mf1x00000gp/T/repro-8UTZo4N/ds000003-qc/workdir/mriqc_wf/anatMRIQC/AFNISkullStripWorkflow/_in_file_..var..folders..tk..5_lnk94x727gwq_x4z6mf1x00000gp..T..repro-8UTZo4N..ds000003-qc..sourcedata..sub-13..anat..sub-13_T1w.nii.gz/binarize/sub-13_T1w_conformed_calc_mask.nii.gz --transform /var/folders/tk/5_lnk94x727gwq_x4z6mf1x00000gp/T/repro-8UTZo4N/ds000003-qc/workdir/mriqc_wf/anatMRIQC/SpatialNormalization/_in_file_..var..folders..tk..5_lnk94x727gwq_x4z6mf1x00000gp..T..repro-8UTZo4N..ds000003-qc..sourcedata..sub-13..anat..sub-13_T1w.nii.gz/SpatialNormalization/ants_t1_to_mniInverseComposite.h5
Standard output:

Standard error:
Killed
Return code: 137

Traceback (most recent call last):
  File "/usr/local/miniconda/bin/mriqc", line 10, in <module>
    sys.exit(main())
  File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/cli/run.py", line 69, in main
    mriqc_wf.run(**config.nipype.get_plugin())
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/workflows.py", line 632, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/base.py", line 164, in run
    self._clean_queue(jobid, graph, result=result)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/base.py", line 227, in _clean_queue
    raise RuntimeError("".join(result["traceback"]))
RuntimeError: Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 434, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 829, in _run_interface
    self.raise_exception(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 760, in raise_exception
    ).format(**runtime.dictcopy())
RuntimeError: Command:
antsApplyTransforms --default-value 0 --dimensionality 3 --float 1 --input /home/bidsapp/.cache/templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-01_desc-head_mask.nii.gz --interpolation MultiLabel --output tpl-MNI152NLin2009cAsym_res-01_desc-head_mask_trans.nii.gz --reference-image /var/folders/tk/5_lnk94x727gwq_x4z6mf1x00000gp/T/repro-8UTZo4N/ds000003-qc/workdir/mriqc_wf/anatMRIQC/AFNISkullStripWorkflow/_in_file_..var..folders..tk..5_lnk94x727gwq_x4z6mf1x00000gp..T..repro-8UTZo4N..ds000003-qc..sourcedata..sub-13..anat..sub-13_T1w.nii.gz/binarize/sub-13_T1w_conformed_calc_mask.nii.gz --transform /var/folders/tk/5_lnk94x727gwq_x4z6mf1x00000gp/T/repro-8UTZo4N/ds000003-qc/workdir/mriqc_wf/anatMRIQC/SpatialNormalization/_in_file_..var..folders..tk..5_lnk94x727gwq_x4z6mf1x00000gp..T..repro-8UTZo4N..ds000003-qc..sourcedata..sub-13..anat..sub-13_T1w.nii.gz/SpatialNormalization/ants_t1_to_mniInverseComposite.h5
Standard output:

Standard error:
Killed
Return code: 137

[INFO   ] == Command exit (modification check follows) =====
run(error): /var/folders/tk/5_lnk94x727gwq_x4z6mf1x00000gp/T/repro-8UTZo4N/ds000003-qc (dataset) [./containers/scripts/singularity_cmd run...]
action summary:
  get (notneeded: 2, ok: 7)
  run (error: 1)

DataLad 1.0.2 WTF (system)

WTF

system

• distribution: darwin/20.3.0 10.16/x86_64
• encoding:
  • default: utf-8
  • filesystem: utf-8
  • locale.prefered: UTF-8
• filesystem:
  • CWD:
    • max_pathlength: 1024
    • mount_opts: ro,local,rootfs,dovolfs,journaled,multilabel
    • path: /Users/yoh/proj/repronim/containers
    • type: apfs
  • HOME:
    • max_pathlength: 1024
    • mount_opts: ro,local,rootfs,dovolfs,journaled,multilabel
    • path: /Users/yoh
    • type: apfs
  • TMP:
    • max_pathlength: 1024
    • mount_opts: ro,local,rootfs,dovolfs,journaled,multilabel
    • path: /var/folders/tk/5_lnk94x727gwq_x4z6mf1x00000gp/T
    • type: apfs
• max_path_length: 291
• name: Darwin
• release: 20.3.0
• type: posix
• version: Darwin Kernel Version 20.3.0: Thu Jan 21 00:07:06 PST 2021; root:xnu-7195.81.3~1/RELEASE_X86_64

@djarecka @satra what would happen on your mac if you do

git clone https://github.com/ReproNim/containers
cd containers
bash <(sed -n -e '/^ *#!/,/^```$/p' README.md | grep -v '```')

? (@dnkennedy try doing that exact set of commands too for consistency)

did you may be observe that

File "/usr/local/miniconda/lib/python3.7/multiprocessing/forkserver.py", line 64, in connect_to_new_process
self.ensure_running()
File "/usr/local/miniconda/lib/python3.7/multiprocessing/forkserver.py", line 118, in ensure_running
listener.bind(address)
OSError: [Errno 22] Invalid argument

before under docker on macs?

[mgxd]

Also tested on a mac, and seeing a different issue with multiprocessing

240620-07:50:10,370 cli IMPORTANT:
	 
    Running MRIQC version 0.16.0:
      * BIDS dataset path: /var/folders/r6/mcdps6rs37x2ttd1fn032yy40000gp/T/repro-AaUAJmz/ds000003-qc/sourcedata.
      * Output folder: /var/folders/r6/mcdps6rs37x2ttd1fn032yy40000gp/T/repro-AaUAJmz/ds000003-qc.
      * Analysis levels: ['group', 'participant'].

You are using MRIQC v0.16.0, and a newer version is available: 24.0.0.
Traceback (most recent call last):
  File "/usr/local/miniconda/bin/mriqc", line 10, in <module>
    sys.exit(main())
  File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/cli/run.py", line 37, in main
    with Manager() as mgr:
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/context.py", line 56, in Manager
    m.start()
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/managers.py", line 543, in start
    self._process.start()
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/context.py", line 291, in _Popen
    return Popen(process_obj)
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/popen_forkserver.py", line 35, in __init__
    super().__init__(process_obj)
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/popen_forkserver.py", line 51, in _launch
    self.sentinel, w = forkserver.connect_to_new_process(self._fds)
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/forkserver.py", line 64, in connect_to_new_process
    self.ensure_running()
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/forkserver.py", line 119, in ensure_running
    os.chmod(address, 0o600)
OSError: [Errno 22] Invalid argument: '/tmp/pymp-e12c8p4i/listener-24orxuis'
[INFO   ] == Command exit (modification check follows) ===== 
run(error): /var/folders/r6/mcdps6rs37x2ttd1fn032yy40000gp/T/repro-AaUAJmz/ds000003-qc (dataset) [./code/containers/scripts/singularity_cm...]
action summary:
  get (notneeded: 2, ok: 7)
  run (error: 1)

I vaguely remember a multiprocessing bug in 3.7 but cannot quickly find anything - @oesteban ring any bells?

[dnkennedy]

Yarik hoped the new version would work. It didn't.

output of running

% datalad containers-run \
        -n bids-mriqc \
        --input sourcedata \
        --output . \
        '{inputs}' '{outputs}' participant group -w workdir
[INFO   ] Making sure inputs are available (this may take some time) 
get(ok): sourcedata/sub-02/anat/sub-02_T1w.nii.gz (file) [from s3-PUBLIC...]                                                                        
get(ok): sourcedata/sub-02/anat/sub-02_inplaneT2.nii.gz (file) [from s3-PUBLIC...]                                                                  
get(ok): sourcedata/sub-02/func/sub-02_task-rhymejudgment_bold.nii.gz (file) [from s3-PUBLIC...]
get(ok): sourcedata/sub-13/anat/sub-13_T1w.nii.gz (file) [from s3-PUBLIC...]
get(ok): sourcedata/sub-13/anat/sub-13_inplaneT2.nii.gz (file) [from s3-PUBLIC...]
get(ok): sourcedata/sub-13/func/sub-13_task-rhymejudgment_bold.nii.gz (file) [from s3-PUBLIC...]
get(ok): code/containers/images/bids/bids-mriqc--24.0.0.sing (file) [from origin...]                                                                
[INFO   ] == Command start (output follows) =====                                                                                                   
2024-06-21 05:06:43 | WARNING  | matplotlib       | Matplotlib created a temporary cache directory at /tmp/matplotlib-pmaf7zp9 because the default path (/home/mriqc/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
------------------------------------------------------------------
  Running MRIQC version 24.1.0.dev0+g3fe90466.d20240417
  ----------------------------------------------------------------
  * BIDS dataset path: /Users/kennedyd/my-experiments/ds000003-qc/sourcedata.
  * Output folder: ..
  * Analysis levels: ['participant', 'group'].
------------------------------------------------------------------

Process SyncManager-1:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/lib/python3.11/multiprocessing/managers.py", line 592, in _run_server
    server = cls._Server(registry, address, authkey, serializer)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/multiprocessing/managers.py", line 156, in __init__
    self.listener = Listener(address=address, backlog=128)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/multiprocessing/connection.py", line 464, in __init__
    self._listener = SocketListener(address, family, backlog)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/multiprocessing/connection.py", line 607, in __init__
    self._socket.bind(address)
OSError: [Errno 22] Invalid argument
Traceback (most recent call last):
  File "/opt/conda/bin/mriqc", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/mriqc/cli/run.py", line 104, in main
    with Manager() as mgr:
         ^^^^^^^^^
  File "/opt/conda/lib/python3.11/multiprocessing/context.py", line 57, in Manager
    m.start()
  File "/opt/conda/lib/python3.11/multiprocessing/managers.py", line 567, in start
    self._address = reader.recv()
                    ^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
          ^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/multiprocessing/connection.py", line 430, in _recv_bytes
    buf = self._recv(4)
          ^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/multiprocessing/connection.py", line 399, in _recv
    raise EOFError
EOFError
[INFO   ] == Command exit (modification check follows) ===== 
run(error): /Users/kennedyd/my-experiments/ds000003-qc (dataset) [./code/containers/scripts/singularity_cm...]
add(ok): .bids_db/layout_index.sqlite (file)                                                                                                        
add(ok): logs/mriqc-20240621-050643_0195cf7d-f0f5-4206-a443-09895e75f460.log (file)                                                                 
save(ok): . (dataset)                                                                                                                               
action summary:                                                                                                                                     
  add (ok: 2)
  get (notneeded: 2, ok: 7)
  run (error: 1)
  save (notneeded: 2, ok: 1)