Running in cluster

[angelicagallegonar]

Does someone now if it is possible to paralelize between different nodes in HybPiper? Some of my read files are huge and cannot be processed with the 24 cores available per node in the cluster I´m using. At the exonerate step it stops due to a lack of memory. This is the code I´ve been using but it isn´t paralelizing between nodes:

#!/bin/bash
#SBATCH -p special
#SBATCH -N 8
#SBATCH -n 40
#SBATCH -e HybPiper27.out
#SBATCH -o HybPiper27.err

cd /home/rjb/mfmazuecos/HybSeq_AGN/HybPiper_reads_first
hybpiper assemble -t_dna Araliaceae.fasta -r Met_dav_R* --prefix Met_dav_S31REV --bwa --cpu 40

[mossmatters]

Unfortunately, HybPiper does not have parallel capabilities, it can only run multithreaded on a single node. The reads are not used at the Exonerate step, so the memory error is likely from running HybPiper with --cpu 40 on a node with only 24 cores.

If you were to run with 24 cores, where does your slowdown come from? The read-mapping (BWA) stage, the distribute stage, or the assemble (spades) stage?

If you have a log file you can share, I would be happy to take a look.