Testing various workflow engines on AWS Batch and Slurm (for now; perhaps further testing to come on GCP and Azure).
Some researchers at the Hutch are ready for more robust workflow engines to use with either Slurm or AWS Batch. Therefore we should be evaluating engines which claim to work on both back ends.
Here are the workflow engines that were evaluated:
| Engine | Source | Written in | Workflow language(s) | Actively Maintained | Slurm support | AWS Batch Support | GCP Support | Azure Support | Notes |
|---|---|---|---|---|---|---|---|---|---|
| Cromwell | Broad Institute | Scala | CWL/WDL | ☑️ | ☑️ | ☑️ | ☑️ | ❌ | 1 |
| NextFlow | Center for Genomic Regulation (Spain) | Java | Groovy DSL | ☑️ | ☑️ | ☑️ | ❌ | ❌ | 2 |
| Toil | UCSC | Python | Python/CWL | ☑️ | ☑️ | ❌ | ☑️ | ☑️ | 3 |
| Luigi | Spotify | Python | Python | ☑️ | ❌ | ☑️ | ☑️ | ❌ | |
| SciLuigi | Pharmbio (SE) / jgolob | Python | Python | ❌ | ☑️ | ☑️ | via luigi? | ❌ | 4 |
1 GCP support via Google Genomics Pipelines API.
2 Has Kubernetes support so that may allow it to work on GCP.
3 Runs on AWS (EC2, ECS(?), but not Batch).
4 jgolob's fork, container-task branch provides AWS Batch support, he and sminot are actively using it, but the original maintainers
have moved on to another project and would be willing to hand this project over to a new maintainer.
In this repository you should find a directory for each engine,
each of which contains batch and slurm directories. In each you'll find a README with instructions and links, and files needed to run the examples.
These examples assume you have administrator or otherwise high-level access
to the SciComp AWS account.