You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
initial EC2 initialization and installation of datalad and datalad-container
(git)lena:~/proj/repronim/reproman[master]git
$> reproman -l debug create -t aws-ec2 my-nitrc-quick -b instance_type=t2.medium -b image=ami-0acbd99fe8c84efbb
You did not specify an EC2 SSH key-pair name to use when creating your EC2
environment.
Please enter a unique name to create a new key-pair or press [enter] to exit: my-nitrc-quick
That key name exists already, try again: my-nitrc-quick3
2020-12-15 08:35:02,531 [INFO ] Created private key file /home/yoh/.local/share/reproman/ec2_keys/my-nitrc-quick3.pem
2020-12-15 08:35:05,205 [INFO ] Waiting for EC2 instance i-04cc7bfb6534fe133 to start running...
2020-12-15 08:36:06,037 [INFO ] EC2 instance i-04cc7bfb6534fe133 is running!
2020-12-15 08:36:06,037 [INFO ] Waiting for EC2 instance i-04cc7bfb6534fe133 to complete initialization...
2020-12-15 08:38:52,706 [INFO ] EC2 instance i-04cc7bfb6534fe133 initialized!
2020-12-15 08:38:52,729 [INFO ] Created the environment my-nitrc-quick
$> reproman login my-nitrc-quick
ubuntu@nitrcce:~$ sudo apt-get update -qqq
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 InRelease: The following signatures were invalid: EXPKEYSIG 6ED91CA3AC1160CD NVIDIA CORPORATION (Open Source Projects) <[email protected]>
W: Failed to fetch https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64/InRelease The following signatures were invalid: EXPKEYSIG 6ED91CA3AC1160CD NVIDIA CORPORATION (Open Source Projects) <[email protected]>
W: Some index files failed to download. They have been ignored, or old ones used instead.
ubuntu@nitrcce:~$ sudo apt-get install -y datalad datalad-container
Reading package lists... Done
...
Get:7 http://neuro.debian.net/debian bionic/main amd64 datalad all 0.13.6-1~nd18.04+1 [137 kB]
Get:8 http://neuro.debian.net/debian bionic/main amd64 python3-datalad all 0.13.6-1~nd18.04+1 [1109 kB]
Get:9 http://neuro.debian.net/debian bionic/main amd64 datalad-container all 1.1.0-1~nd18.04+1 [26.2 kB]
Get:10 http://neuro.debian.net/debian bionic/main amd64 singularity-container amd64 2.6.1-2+nd2~nd18.04+1 [351 kB]
.... took awhile -- that instance is slow :-/ ...
configuration and failed execution
ubuntu@nitrcce:~$ datalad install ///repronim/containers
It is highly recommended to configure Git before using DataLad. Set both 'user.name' and 'user.email' configuration variables.
It is highly recommended to configure Git before using DataLad. Set both 'user.name' and 'user.email' configuration variables.
[INFO ] *** Please tell me who you are.
|
| Run
|
| git config --global user.email "[email protected]"
| git config --global user.name "Your Name"
|
| to set your account's default identity.
| Omit --global to set the identity only in this repository.
|
| fatal: unable to auto-detect email address (got 'ubuntu@nitrcce.(none)')
install(ok): /home/ubuntu/containers (dataset)
ubuntu@nitrcce:~$ git config --global user.email "[email protected]"
ubuntu@nitrcce:~$ git config --global user.name "Your Name"
ubuntu@nitrcce:~$ cd containers/
ubuntu@nitrcce:~/containers$ bash <(sed -n -e '/^ *#!/,/^```$/p' README.md | grep -v '```')
...
> datalad containers-run -n containers/bids-mriqc --input sourcedata --output . '{inputs}' '{outputs}' participant group
[INFO ] Making sure inputs are available (this may take some time)
get(ok): sourcedata/sub-02/anat/sub-02_T1w.nii.gz (file) [from s3-PUBLIC...]
get(ok): sourcedata/sub-02/anat/sub-02_inplaneT2.nii.gz (file) [from s3-PUBLIC...]
get(ok): sourcedata/sub-02/func/sub-02_task-rhymejudgment_bold.nii.gz (file) [from s3-PUBLIC...]
get(ok): sourcedata/sub-13/anat/sub-13_T1w.nii.gz (file) [from s3-PUBLIC...]
get(ok): sourcedata/sub-13/anat/sub-13_inplaneT2.nii.gz (file) [from s3-PUBLIC...]
get(ok): sourcedata/sub-13/func/sub-13_task-rhymejudgment_bold.nii.gz (file) [from s3-PUBLIC...]
get(ok): containers/images/bids/bids-mriqc--0.15.1.sing (file) [from origin...]
[INFO ] == Command start (output follows) =====
2020-12-15 15:35:23,242 mriqc:IMPORTANT
Running MRIQC version 0.15.1:
* BIDS dataset path: /tmp/repro-3vNHTmV/ds000003-qc/sourcedata.
* Output folder: /tmp/repro-3vNHTmV/ds000003-qc.
* Analysis levels: participant, group.
Process Process-2:
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/bin/mriqc_run.py", line 400, in init_mriqc
ignore=['derivatives', 'sourcedata', r'^\..*'])
File "/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/layout.py", line 185, in __init__
self._validate_root()
File "/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/layout.py", line 318, in _validate_root
raise ValueError("BIDS root does not exist: %s" % self.root)
ValueError: BIDS root does not exist: /tmp/repro-3vNHTmV/ds000003-qc/sourcedata
[INFO ] == Command exit (modification check follows) =====
[INFO ] The command had a non-zero exit code. If this is expected, you can save the changes with 'datalad save -d . -r -F .git/COMMIT_EDITMSG'
CommandError: 'containers/scripts/singularity_cmd run containers/images/bids/bids-mriqc--0.15.1.sing 'sourcedata' '' participant group' failed with exitcode 1 under /tmp/repro-3vNHTmV/ds000003-qc
more specific reproducing + not-reproducing with direct invocation when already in the container
ubuntu@nitrcce:/tmp/repro-3vNHTmV/ds000003-qc$ singularity exec -W /tmp/singtmp.PacrEs -B /tmp/singtmp.PacrEs/tmp:/tmp -B /tmp/singtmp.PacrEs/var/tmp:/var/tmp -e -B /tmp/repro-3vNHTmV/ds000003-qc -H /tmp/repro-3vNHTmV/ds000003-qc/containers/binds/HOME --pwd /tmp/repro-3vNHTmV/ds000003-qc containers/images/bids/bids-mriqc--0.15.1.sing bash
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
bidsapp@nitrcce:/tmp/repro-3vNHTmV/ds000003-qc$ export| grep SING
declare -x SINGULARITY_APPNAME=""declare -x SINGULARITY_CONTAINER="/tmp/repro-3vNHTmV/ds000003-qc/containers/images/bids/bids-mriqc--0.15.1.sing"declare -x SINGULARITY_NAME="bids-mriqc--0.15.1.sing"
bidsapp@nitrcce:/tmp/repro-3vNHTmV/ds000003-qc$ ls /tmp/repro-3vNHTmV/ds000003-qc/sourcedata
CHANGES README dataset_description.json participants.tsv sub-02 sub-13 task-rhymejudgment_bold.json
bidsapp@nitrcce:/tmp/repro-3vNHTmV/ds000003-qc$ /.singularity.d/runscript sourcedata '' participant group
2020-12-15 15:40:41,456 mriqc:IMPORTANT
Running MRIQC version 0.15.1:
* BIDS dataset path: /tmp/repro-3vNHTmV/ds000003-qc/sourcedata.
* Output folder: /tmp/repro-3vNHTmV/ds000003-qc.
* Analysis levels: participant, group.
201215-15:40:47,656 nipype.workflow INFO:
Building anatomical MRIQC workflow, datasets list: ['sub-02/anat/sub-02_T1w.nii.gz', 'sub-13/anat/sub-13_T1w.nii.gz']
2020-12-15 15:40:47,656 nipype.workflow:INFO Building anatomical MRIQC workflow, datasets list: ['sub-02/anat/sub-02_T1w.nii.gz', 'sub-13/anat/sub-13_T1w.nii.gz']
^CException ignored in: <bound method InstanceState._cleanup of <sqlalchemy.orm.state.InstanceState object at 0x7f6eeb461048>>
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 399, in _cleanup
del self._instance_dict
KeyboardInterrupt
Traceback (most recent call last):
File "/usr/local/miniconda/bin/mriqc", line 10, in<module>sys.exit(main())
File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/bin/mriqc_run.py", line 221, in main
plugin_settings = retval['plugin_settings']
File "<string>", line 2, in __getitem__
File "/usr/local/miniconda/lib/python3.7/multiprocessing/managers.py", line 795, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/usr/local/miniconda/lib/python3.7/multiprocessing/connection.py", line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/usr/local/miniconda/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
self._send(header + buf)
File "/usr/local/miniconda/lib/python3.7/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
I Ctrl-C'ed above run since it did start going
so it feels like bind mount of /tmp/repro-3vNHTmV/ds000003-qc did not happen... but reason seems to be somewhere due to our singularity_cmd halper since it fails with it:
it fails with it
ubuntu@nitrcce:/tmp/repro-3vNHTmV/ds000003-qc$ bash -x containers/scripts/singularity_cmd run containers/images/bids/bids-mriqc--0.15.1.sing 'sourcedata'. participant group
+ set -eu
++ readlink -f containers/scripts/singularity_cmd
+ thisfile=/tmp/repro-3vNHTmV/ds000003-qc/containers/scripts/singularity_cmd
++ dirname /tmp/repro-3vNHTmV/ds000003-qc/containers/scripts/singularity_cmd
+ thisdir=/tmp/repro-3vNHTmV/ds000003-qc/containers/scripts
++ dirname /tmp/repro-3vNHTmV/ds000003-qc/containers/scripts
+ updir=/tmp/repro-3vNHTmV/ds000003-qc/containers
+ BHOME=/tmp/repro-3vNHTmV/ds000003-qc/containers/binds/HOME
+ cmd=run
+ shift
+ '[' -n ''']'
++ mktemp -d -t singtmp.XXXXXX
+ tmpdir=/tmp/singtmp.ywiVAe
+ info 'created temp dir /tmp/singtmp.ywiVAe'
+ :
+ trap'rm -fr "$tmpdir" && info "removed temp dir $tmpdir"'exit
+ pass_git_config user.name 'ReproNim User'
+ var=user.name
+ default='ReproNim User'
++ git config user.name
+ value='Your Name'
+ forattemptin {1..5}
+ git config -f /tmp/repro-3vNHTmV/ds000003-qc/containers/binds/HOME/.gitconfig user.name
+ break
+ pass_git_config user.email [email protected]
+ var=user.email
+ [email protected]
++ git config user.email
+ [email protected]
+ forattemptin {1..5}
+ git config -f /tmp/repro-3vNHTmV/ds000003-qc/containers/binds/HOME/.gitconfig user.email
+ break
+ SARGS=(-e -B "$PWD" -H "$BHOME" --pwd "$PWD""$@")
+ need_no_c=
+ fordin"$PWD""$updir"
+ '[' repro-3vNHTmV/ds000003-qc '!=' /tmp/repro-3vNHTmV/ds000003-qc ']'
+ info 'Creating /tmp/repro-3vNHTmV/ds000003-qc under /tmp/singtmp.ywiVAe'
+ :
+ mkdir -p /tmp/singtmp.ywiVAe//tmp/repro-3vNHTmV/ds000003-qc /tmp/singtmp.ywiVAe/var/tmp
+ need_no_c=1
+ fordin"$PWD""$updir"
+ '[' repro-3vNHTmV/ds000003-qc/containers '!=' /tmp/repro-3vNHTmV/ds000003-qc/containers ']'
+ info 'Creating /tmp/repro-3vNHTmV/ds000003-qc/containers under /tmp/singtmp.ywiVAe'
+ :
+ mkdir -p /tmp/singtmp.ywiVAe//tmp/repro-3vNHTmV/ds000003-qc/containers /tmp/singtmp.ywiVAe/var/tmp
+ need_no_c=1
+ '[' -z 1 ']'
+ SARGS=(-B "$tmpdir/tmp:/tmp" -B "$tmpdir/var/tmp:/var/tmp""${SARGS[@]}")
+ hash singularity
+ '[' -z ''']'
+ singularity run -W /tmp/singtmp.ywiVAe -B /tmp/singtmp.ywiVAe/tmp:/tmp -B /tmp/singtmp.ywiVAe/var/tmp:/var/tmp -e -B /tmp/repro-3vNHTmV/ds000003-qc -H /tmp/repro-3vNHTmV/ds000003-qc/containers/binds/HOME --pwd /tmp/repro-3vNHTmV/ds000003-qc containers/images/bids/bids-mriqc--0.15.1.sing sourcedata . participant group
2020-12-15 15:46:55,399 mriqc:IMPORTANT
Running MRIQC version 0.15.1:
* BIDS dataset path: /tmp/repro-3vNHTmV/ds000003-qc/sourcedata.
* Output folder: /tmp/repro-3vNHTmV/ds000003-qc.
* Analysis levels: group, participant.
Process Process-2:
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/bin/mriqc_run.py", line 400, in init_mriqc
ignore=['derivatives', 'sourcedata', r'^\..*'])
File "/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/layout.py", line 185, in __init__
self._validate_root()
File "/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/layout.py", line 318, in _validate_root
raise ValueError("BIDS root does not exist: %s" % self.root)
ValueError: BIDS root does not exist: /tmp/repro-3vNHTmV/ds000003-qc/sourcedata
+ rm -fr /tmp/singtmp.ywiVAe
+ info 'removed temp dir /tmp/singtmp.ywiVAe'
+ :
but works if I just go with the direct singularity command it runs:
so the reason is really is having that /tmp/singtmp.ywiVAe (and subdirs) which bind mount first and then elderly singularity refuses to bind mount /tmp/repro-3vNHTmV/ds000003-qc on top of it...
quick&dirty workaround is to place TMPDIR somewhere else but /tmp, e.g. mkdir -p ~/tmp and provide it while running the example, e.g. TMPDIR=~/tmp bash <(sed -n -e '/^ *#!/,/^```$/p' README.md | grep -v '```')
then we do not bother to create/bind mount those additional paths
initial EC2 initialization and installation of datalad and datalad-container
configuration and failed execution
more specific reproducing + not-reproducing with direct invocation when already in the container
I Ctrl-C'ed above run since it did start going
so it feels like bind mount of
/tmp/repro-3vNHTmV/ds000003-qcdid not happen... but reason seems to be somewhere due to our singularity_cmd halper since it fails with it:it fails with it
but works if I just go with the direct singularity command it runs:
so the reason is really is having that
/tmp/singtmp.ywiVAe(and subdirs) which bind mount first and then elderly singularity refuses to bind mount /tmp/repro-3vNHTmV/ds000003-qc on top of it...quick&dirty workaround is to place TMPDIR somewhere else but
/tmp, e.g.mkdir -p ~/tmpand provide it while running the example, e.g.TMPDIR=~/tmp bash <(sed -n -e '/^ *#!/,/^```$/p' README.md | grep -v '```')then we do not bother to create/bind mount those additional paths
proper fix is yet to be worked out...
The text was updated successfully, but these errors were encountered: