"Screw up at restart" issue

[illdopejake]

Hey all, reporting here an issue I had when preprocessing ~100 ADPD subjects using Niak 13.4b. I saw that the pipeline was still running but several of my workers were idle. Output said this:

Also, this from the daemon logs:

Looking in logs/workers, psom2-4 (out of 5) were empty. In psom1, PB pointed out that there was .ready file but no new_jobs.mat file, which apparently can happen if a worker is restarted exactly when the manager assigns it new jobs. But, number of worker deaths was puzzling.

psom5 was the only directory with worker.eqsub and worker.oqsub files. They read as follows:

Path to output: /gs/project/gsf-624-aa/ADPD_October_2015/preproc/fmri_preprocess_all_scrubb05/

Path to script: /gs/project/gsf-624-aa/ADPD_October_2015/scripts/nkim_preprocessing_jake.m

In solidarity,
--Jake