Possibility of native osx-arm64 support for rotary

[jmtsuji]

Following up from #228, the purpose of this issue is to assess how difficult it would be to add native osx-arm64 install support for all of rotary's conda environments. Based on this assessment, we can decide if it would make sense to pursue native osx-arm64 install support for rotary.

Current conda envs and osx-arm64 install compatibility on conda:

• ⚠️ annotation_dfast.yaml: DFAST dependencies blast, aragorn, and possibly others lack a osx-arm64 install
• ✅ assembly_flye.yaml OK
• ❌ checkm2.yaml: DIAMOND lacks a osx-arm64 install
• ❌ circlator.yaml: circulator dependencies (e.g., canu) lack osx-arm64 install support. However, we plan to eventually swap out circlator for spokewrench.
• ✅ download.yaml OK
• ❌ eggnog.yaml: the eggnog-mapper dependency prodigal has no osx-arm64 install support. This could potentially be swapped out with pyrodigal, which has osx-arm64 support. DIAMOND also has no osx-arm64 install support.
• ❌ gtdbtk.yaml: the GTDB-Tk dependencies mash, pplacer, prodigal, and skani lack osx-arm64 install support
• ⚠️ mapping.yaml: bwa-mem2 and prodigal lack osx-arm64 install support. We could change these for bwa and pyrodigal, which have osx-arm64 installs, if needed.
• ⚠️ medaka.yaml: OK if we upgrade to medaka v2 (see medaka on macOS #228)
• ✅ polypolish.yaml OK
• ❌ pypolca.yaml: freebayes lacks osx-arm64 install support
• ❌ qc.yaml: QUAST depends on blast, which does not have a osx-arm64 conda install.
• ❌ spokewrench.yaml: mummer4 lacks a osx-arm64 conda install. circlator also lacks a osx-arm64 install, but we plan to eventually drop circlator as a dependency of spokewrench.

I have tested some but not all of these during an actual install on a M2 Mac. The others were just evaluated roughly by looking through the first layer or two of their dependencies on the conda website.

The above situation does not look promising... I do not think it would be feasible for us to ask developers (or conda maintainers) of all the dependencies mentioned above to make osx-arm64 conda installs. I also do not think it would be feasible for us to make osx-arm64 installs of these ourselves, given the large number of tools and time constraints on our end. We could consider choosing alternative tools in some cases (e.g., replace DFAST with a different tool), but I think we cannot do this in all cases.

Thus, I propose that we wait on making rotary directly installable via osx-arm64. The situation might improve in the future as more osx-arm64 installs are made on conda. @LeeBergstrand What are your thoughts?

[LeeBergstrand]

@jmtsuji Going from Intel to ARM architecture is going to be a big jump for most bioinformatics software. I doubt many of the software's involved is going to even have linux-arm64 compatibility which would be the baseline before doing the osx-arm64 version. Many of the software's in bioinformatics were written for X86-64 (intel / amd) architecture with no expectation of running on ARM, which is still mostly used on phones. ARM64 server architecture is very rare. There is only one or two companies that make ARM rack servers:

https://www.gigabyte.com/Industry-Solutions/ampere-altra-server-solution

And Google cloud has ARM VMs. https://cloud.google.com/compute/docs/instances/arm-on-compute

So usually companies take an existing pipeline that runs on X86-64 and then modify it to run on ARM with huge corporate investment. ARM draws less electricity than Intel per compute amount so it's cheaper to run at scale.

Also there are certain tools like bwa-mem2 that were optimized with the help of Intel's RnD team to use certain CPU intel instruction to be faster and thus can never be ported to ARM.

The original bwa was developed by Heng Li (@lh3). Performance enhancement in bwa-mem2 was primarily done by Vasimuddin Md (@yuk12) and Sanchit Misra (@sanchit-misra) from Parallel Computing Lab, Intel.

[LeeBergstrand]

Thus, I propose that we wait on making rotary directly installable via osx-arm64. The situation might improve in the future as more osx-arm64 installs are made on conda. @LeeBergstrand What are your thoughts?

I agree with this. I don't think OSX support should be pursued at this time. Considering the RAM requirements of the software there won't be too many mac that can run the full pipeline either.

[LeeBergstrand]

@jmtsuji The other issue is for conda to support ARM packages the package has to be built on an ARM machine.

Bioconda didn't even support ARM builds until a few month ago: https://bioconda.github.io/developer/aarch64.html

Also see: https://bioconda.github.io/faqs.html#platform-nomenclature-faq

bioconda/bioconda-recipes#23454

I'm going to assume that in a year or two most of the software in the pipeline other than tools that were intel directly assisted in the development (e.g. bwa-mem2) will be ported over to ARM but we will have to wait.

[LeeBergstrand]

mapping.yaml: bwa-mem2 and prodigal lack osx-arm64 install support. We could change these for bwa and pyrodigal, which have osx-arm64 installs, if needed.

Though harder to port. They are looking at ways to port BWA mem2 bwa-mem2/bwa-mem2#206 (comment)

[jmtsuji]

@LeeBergstrand Thanks for the additional background. Yeah, I also saw that bioconda only recently added osx-arm64 support. Good to know that some tools like BWA mem2 might be quite challenging to port. Let's wait on this and potentially revisit it in the future, as discussed in #228

[jmtsuji]

Minor update: DIAMOND now has a osx-arm64 install from version 2.1.10:
https://x.com/bbuchfink/status/1848272144171495616
https://anaconda.org/bioconda/diamond