Working HPC - is there a docker version, singularity version

[soilmicrobiome]

Hi there,

I am working on a HPC that we are not permitted to use conda on. Is there instruction to compile from scratch or a docker or singularity version available?

Thanks

[SilasK]

Oh, this is unfortunate. It won't allow you to access other nodes on your cluster.

Here would be the instructions to use Docker.
https://github.com/metagenome-atlas/atlas/blob/a744644956ab143b1a0005f6c6022a0e027fcf8c/docs/usage/docker.rst

The atlas verison in the Docker container are V2.3

If you want you to rebuild the docker file it uses the old one and updates atlas: https://github.com/metagenome-atlas/atlas/blob/master/Dockerfile

[soilmicrobiome]

I am also working on a HPC that does not allow conda. But i found when i use:

$singularity pull metagenomeatlas/atlas

I get the following message:
"FATAL: While pulling library image: error fetching image: image does not exist in the library: metagenomeatlas/atlas:latest (amd64)"

Any ideas?

[SilasK]

Hmm, I should find a solution for this use case.

Is there a container with miniconda or mambaforge for singularity?

https://github.com/conda-forge/miniforge#mambaforge

[soilmicrobiome]

I could not see if this was the case. And on the HPC i use there is not option for docker. any ideas on what i can do?

[SilasK]

Ok, @soilmicrobiome

My solution would be as follows:

1. Install all dependencies for atlas.

   • Except for bbmap all others are python packages and should be on pip.
   • Install or load singularity.

2. Install atlas.

   • For now, install metagenome atlas from Github (see docs). The changes e54a76e are not yet in the conda version.

3. Run atlas with the additional argument --use-singularity

   • I also recommend to use --singularity-prefix /path/to/store/syngularity/packages and if you need --singularity-args

atlas should now create each conda env in a singularity container.

The important advantage this solution is compatible with cluster execution. E.g. It is possible to let atlas submit multiple jobs to your cluster system. See the docs for more info.

[soilmicrobiome]

great! i'll give it a try

[soilmicrobiome]

I finally got a version of atlas working with singularity (apparently one of our servers has a file system that is not compatible with the container, but another one was). I can successfully run "atlas init". But when it comes to "atlas run all" there are still conda issues:

part of the error message:

_[2021-05-15 11:22 INFO] Executing: snakemake --snakefile /cvmfs/HPC_name/easybuild/software/2020/avx2/Compiler/intel2020/metagenome-atlas/2.4.3/lib/python3.6/site-packages/atlas/Snakefile --directory /project/6010084/user/path_to_files --rerun-incomplete --configfile '/project/6010084/user/path_to_files/config.yaml' --nolock --use-conda --conda-prefix /project/6010084/databases_software/atlas/databases/conda_envs all

Building DAG of jobs...
Creating conda environment /opt/conda/lib/python3.6/site-packages/atlas/rules/../envs/mmseqs.yaml...
Downloading and installing remote packages.
CreateCondaEnvironmentException:
Could not create conda environment from /opt/conda/lib/python3.6/site-packages/atlas/rules/../envs/mmseqs.yaml:
Collecting package metadata (repodata.json): ...working... failed_

I guess this means the HPC can't create a conda env in a singularity container? Is it time i find a Conda friendly place to run atlas?

[SilasK]

Sorry for the delay. Obviously a conda-frendly version would help. I for example just installed conda in my scratch directory.

Is there no other error about why the conda installation failed?
Often it is an internet connection. Do you think there is a problem with the internet connection within the singularity container?

Maybe an stupid error from my side would be solved by using --conda-frontend conda
You can also use atlas run assembly <other options> --use-singularity --conda-create-envs-only to only create the conda enfs

[soilmicrobiome]

The people that are in charge of the HPC have been working on it over the last few weeks. They think they have it solved -- the last few lines of the output in a test run stated :

"10 of 10 steps (100%) done
Complete log: /scratch/user/consultation/metagenome-atlas/container/.snakemake/log/2021-07-16T205659.541503.snakemake.log
ATLAS finished"

Once i get the details, I'll let you know how they got it to run in case thats of use to anyone!

[SilasK]

Great! Yes, It hink, it would be beneficial to others.

[soilmicrobiome]

they did not give me too much info apart from:

the previous singularity had a "mamba" installed with bugs, which could not create environments. The mamba installed in ubuntu 18.04 works, but not in ubuntu 20.04.

[SilasK]

I wanted to run the metagenome-atlas on an HPC without internet access.

In that case I thought to use the singularity container. I read this issue

Is this singularity image already existing for the new version of metagenome atlas?

Additionally, how to avoid the downloading of the database, which is required for the tool? Can I download the database and keep it inside the singularity image by default?

While there is always an up-to-date docker container for metagenome-atlas it doesn't contain the databases.

I don't think it's a good idea to maintain a 250gb docker container due to several reasons, especially because it doesn't allow to submit multiple jobs to a cluster cloud. I'm also not sure if the public repositories would host such a big container or many versions of it.

I recommend to first explore other options but if you want. to make a docker container here some help:

In the older version, I made a Dockerfile that allows creating such a Docker container, but also it keeps the database outside the docker so the docker can be easily updated.

docker init script

Here Some info from the docs

[SumeetTiwari07]

Thank you for the container and the documentation.

If I understood clearly for each run the database will be downloaded even though the database exists?

[SilasK]

The database is not in the container. I recommend storing the databases outside the docker container via a mounted drive. I don't see how I can create a docker container that includes all the databases.

[SumeetTiwari07]

Sorry for the earlier confusion I do not need the container containing the database inside anymore. Earlier I was under the impression that even if I provide the database externally the tool will again download the whole database from scratch and overwrite the existing one.

[SumeetTiwari07]

I built the singularity image on a VM(ubuntu) as follows:

cmd: singularity build ./metagenomeatlas-2.8.1.sif docker://quay.io/biocontainers/metagenome-atlas:2.8.1--pyhdfd78af_0

1. Then i tried to download the database using the same image on the VM:

   cmd : singularity exec ~/metagenomeatlas-2.8.1.sif atlas download -d ./databases

   Error: "CreateCondaEnvironmentException:
   Could not create conda environment from /usr/local/lib/python3.8/site-packages/atlas/workflow/rules/../envs/gtdbtk.yaml:
   NotWritableError: The current user does not have write permissions to a required path.
   path: /usr/local/pkgs/urls.txt
   uid: 1000
   gid: 1000
   If you feel that permissions on this path are set incorrectly, you can manually
   change them by executing
   $ sudo chown 1000:1000 /usr/local/pkgs/urls.txt
   In general, it's not advisable to use 'sudo conda'."

2. If i tried to download the database on HPC:
   PermissionError: [Errno 13] Permission denied: '.snakemake'

Any suggestions?

[SumeetTiwari07]

Yes, I did that check earlier with the singularity image itself. I am able to see all the --help menus. Even help for specific commands as well.

I do have all the access to that directory.

To see if it's actually the permission issue this I tried on a VM too (where I have complete control). Then I got a totally different error as shown in 1.

[SilasK]

It seems the first error is also related to permissions, see here.

where I have complete control
Does that mean you are root?

Could it be that some files/folders are created by root and you don't have access to them?

Or maybe it's the other way round. the user in the container doesn't have the right to write to the mounted device which is accessible only to the user outside the container?

Going with option 2. What are the folders that you are mounting and what are their access rights inside the container?, Check with ls- l

[SumeetTiwari07]

In option 2:
i just created the directory "~/atlas_db" and trying to download the database there by running this command.
cmd: singularity exec ~/metagenomeatlas-2.8.1.sif atlas download -d ~/atlas_db"
For that directory user(i.e., me) had read/write/x rights where group and others do not have any rights.
One more thing the user doesn't have the sudo rights as well.
Plus if look inside the container:
singularity shell ~/metagenomeatlas-2.8.1.sif then ls -l the user (i.e. root) and groups (i.e. root) have all the rights.

[SilasK]

If I understand correctly your command doesn't mount the folder ~/atlas_db. Is there a way to mount a folder outside the container?

[SumeetTiwari07]

Normally it should be able to mount that directory. But another way of doing that is to use the --bind option. which is blocked by admin on HPC.

Secondly when i tried to run the same on a VM it worked after adding sudo. The command is as follows:
sudo singularity exec ~/metagenomeatlas-2.8.1.sif atlas download -d ~/atlas_db. But i think the container is missing the wget. That i received as next error. Hence it stopped at this rule:
localrule download_atlas_files:
output: /home/ubuntu/atlas_db/adapters.fa
jobid: 1
wildcards: filename=adapters.fa
resources: tmpdir=/tmp

/bin/bash: wget: command not found

[SumeetTiwari07]

Thank you for your help.

1. I have created my own singularity image from scratch and it successfully worked on the VM.
2. For HPC testing, the infrastructure is a bit different here. So I need to understand that first.

Rest everything looks fine. Thank you once again

[SilasK]

Would you willi g tobshate the singularity file, sonothers can benefit from it.

[SumeetTiwari07]

I will be happy to share, after finishing my testing on HPC.

[SilasK]

How did the testing go?

[SumeetTiwari07]

it worked on the VM but not the HPC not able to download database the using the image
since the hpc here does not have internet access.
Since individual step in the pipeline makes a separate conda environment, which didn't work due to lack of internet.
On VM was able to download the the database using the same image.
But I'm hope full that it will work on HPC with internet access.

[SumeetTiwari07]

Hi ,
Here is the link for the singularity image code
https://github.com/SumeetTiwari07/atlas-singularity
It would be nice if someone test it on hpc too. I can see the all the help menus on the HPC with the image.

I have sent an invite too.