LSF profile is broken for newest 7.1.1 (2022-03-07) snakemake version

[leoisl]

Error:

Submitted group job 29137fca-fc8a-5511-88d3-2149b00a8b5e with external jobid '5412741 logs/cluster/group_1/unique/jobid29137fca_07818911-9e6a-460c-a46e-b0f3f0bdb675.out'.
Traceback (most recent call last):
  File "/homes/leandro/.config/snakemake/lsf/lsf_status.py", line 201, in <module>
    jobid = int(sys.argv[1])
ValueError: invalid literal for int() with base 10: '5412741 logs/cluster/group_1/unique/jobid29137fca_07818911-9e6a-460c-a46e-b0f3f0bdb675.out'
WorkflowError:
Failed to obtain job status. See above for error message.

The reason is that on snakemake 7.1.1 the job id is now quoted. LSF profile works on snakemake 7.1.0 though. Will provide a fix soon

[mbhall88]

If you could test this out that would be great @leoisl. Also see #46

[levlitichev]

Thanks for pushing this fix. I can confirm that it worked for me using snakemake v7.3.8.

I'm running into an issue where the default memory limit doesn't seem to be getting pulled from the Cookiecutter profile, and I'm not sure if this is again due to the new snakemake version.

In my Cookiecutter profile (copied at bottom), I set 16 GB as the default memory limit, but this setting doesn't seem to get propagated. When I run a basic rule, the rule uses 1000 MB. But explicitly setting the memory limit to 2000 MB for that rule works fine.

(minimal_snakemake) [litichev@node157 ~]$ cat Snakefile
rule make_file:
    output:
        "new_file.txt"
    shell:
        "touch {output}"
(minimal_snakemake) [litichev@node157 ~]$ snakemake --profile lsf
Building DAG of jobs...
Using shell: /bin/bash
Provided cluster nodes: 500
Job stats:
job          count    min threads    max threads
---------  -------  -------------  -------------
make_file        1              1              1
total            1              1              1

Select jobs to execute...

[Tue Apr 12 21:24:19 2022]
rule make_file:
    output: new_file.txt
    jobid: 0
    resources: mem_mb=1000, disk_mb=1000, tmpdir=/tmp

touch new_file.txt
Submitted job 0 with external jobid '72762310 logs/cluster/make_file/unique/jobid0_07ecd567-f599-4180-863f-9fa2895a48b3.out'.
[Tue Apr 12 21:24:39 2022]
Finished job 0.
1 of 1 steps (100%) done
Complete log: .snakemake/log/2022-04-12T212418.851163.snakemake.log
(minimal_snakemake) [litichev@node157 ~]$ vi Snakefile
(minimal_snakemake) [litichev@node157 ~]$ cat Snakefile
rule make_file:
    output:
        "new_file.txt"
    resources:
        mem_mb=2000
    shell:
        "touch {output}"
(minimal_snakemake) [litichev@node157 ~]$ snakemake --profile lsf --force
Building DAG of jobs...
Using shell: /bin/bash
Provided cluster nodes: 500
Job stats:
job          count    min threads    max threads
---------  -------  -------------  -------------
make_file        1              1              1
total            1              1              1

Select jobs to execute...

[Tue Apr 12 21:25:11 2022]
rule make_file:
    output: new_file.txt
    jobid: 0
    resources: mem_mb=2000, disk_mb=1000, tmpdir=/tmp

touch new_file.txt
Submitted job 0 with external jobid '72762311 logs/cluster/make_file/unique/jobid0_89c6ce42-fe30-453b-923b-32254835bc3e.out'.
[Tue Apr 12 21:25:31 2022]
Finished job 0.
1 of 1 steps (100%) done
Complete log: .snakemake/log/2022-04-12T212511.483596.snakemake.log

My Cookiecutter profile:

(minimal_snakemake) [litichev@node157 ~]$ cat ~/.config/snakemake/lsf/CookieCutter.py
class CookieCutter:
    """
    Cookie Cutter wrapper
    """

    @staticmethod
    def get_default_threads() -> int:
        return int("8")

    @staticmethod
    def get_default_mem_mb() -> int:
        return int("16384")

    @staticmethod
    def get_log_dir() -> str:
        return "logs/cluster"

    @staticmethod
    def get_default_queue() -> str:
        return ""

    @staticmethod
    def get_lsf_unit_for_limits() -> str:
        return "MB"

    @staticmethod
    def get_unknwn_behaviour() -> str:
        return "wait"

    @staticmethod
    def get_zombi_behaviour() -> str:
        return "ignore"

    @staticmethod
    def get_latency_wait() -> float:
        return float("30")

Package versions in my conda environment:

(minimal_snakemake) [litichev@node157 ~]$ conda list
# packages in environment at /home/litichev/miniconda3/envs/minimal_snakemake:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
aioeasywebdav             2.4.0           py39hf3d152e_1001    conda-forge
aiohttp                   3.8.1            py39hb9d737c_1    conda-forge
aiosignal                 1.2.0              pyhd8ed1ab_0    conda-forge
amply                     0.1.4                      py_0    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
async-timeout             4.0.2              pyhd8ed1ab_0    conda-forge
attmap                    0.13.2             pyhd8ed1ab_0    conda-forge
attrs                     21.4.0             pyhd8ed1ab_0    conda-forge
backports                 1.0                        py_2    conda-forge
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
bcrypt                    3.2.0            py39hb9d737c_3    conda-forge
boto3                     1.21.38            pyhd8ed1ab_0    conda-forge
botocore                  1.24.38            pyhd8ed1ab_0    conda-forge
brotlipy                  0.7.0           py39hb9d737c_1004    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2021.10.8            ha878542_0    conda-forge
cachetools                5.0.0              pyhd8ed1ab_0    conda-forge
certifi                   2021.10.8        py39hf3d152e_2    conda-forge
cffi                      1.15.0           py39h4bc2ebd_0    conda-forge
charset-normalizer        2.0.12             pyhd8ed1ab_0    conda-forge
coin-or-cbc               2.10.7               h3786ebc_0    conda-forge
coin-or-cgl               0.60.3               he7e83c3_2    conda-forge
coin-or-clp               1.17.6               h256e9bb_3    conda-forge
coin-or-osi               0.108.6              h3b589db_2    conda-forge
coin-or-utils             2.11.6               h573740c_0    conda-forge
coincbc                   2.10.7            0_metapackage    conda-forge
configargparse            1.5.3              pyhd8ed1ab_0    conda-forge
connection_pool           0.0.3              pyhd3deb0d_0    conda-forge
cryptography              36.0.2           py39hd97740a_1    conda-forge
datrie                    0.8.2            py39h3811e60_3    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
docutils                  0.18.1           py39hf3d152e_1    conda-forge
dropbox                   11.29.0            pyhd8ed1ab_0    conda-forge
filechunkio               1.8                        py_2    conda-forge
filelock                  3.6.0              pyhd8ed1ab_0    conda-forge
frozenlist                1.3.0            py39hb9d737c_1    conda-forge
ftputil                   5.0.3              pyhd8ed1ab_0    conda-forge
gitdb                     4.0.9              pyhd8ed1ab_0    conda-forge
gitpython                 3.1.27             pyhd8ed1ab_0    conda-forge
google-api-core           2.5.0              pyhd8ed1ab_0    conda-forge
google-api-python-client  2.44.0             pyhd8ed1ab_0    conda-forge
google-auth               2.6.3              pyh6c4a22f_0    conda-forge
google-auth-httplib2      0.1.0              pyhd8ed1ab_0    conda-forge
google-cloud-core         2.2.2              pyh6c4a22f_0    conda-forge
google-cloud-storage      2.1.0              pyh6c4a22f_0    conda-forge
google-crc32c             1.1.2            py39hb81f231_2    conda-forge
google-resumable-media    2.1.0              pyh6c4a22f_0    conda-forge
googleapis-common-protos  1.56.0           py39hf3d152e_0    conda-forge
grpcio                    1.45.0           py39h0f497a6_0    conda-forge
httplib2                  0.20.4             pyhd8ed1ab_0    conda-forge
idna                      3.3                pyhd8ed1ab_0    conda-forge
importlib-metadata        4.11.3           py39hf3d152e_1    conda-forge
importlib_resources       5.6.0              pyhd8ed1ab_1    conda-forge
iniconfig                 1.1.1              pyh9f0ad1d_0    conda-forge
jinja2                    3.1.1              pyhd8ed1ab_0    conda-forge
jmespath                  1.0.0              pyhd8ed1ab_0    conda-forge
jsonschema                4.4.0              pyhd8ed1ab_0    conda-forge
jupyter_core              4.9.2            py39hf3d152e_0    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
libblas                   3.9.0           14_linux64_openblas    conda-forge
libcblas                  3.9.0           14_linux64_openblas    conda-forge
libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 11.2.0              h1d223b6_15    conda-forge
libgfortran-ng            11.2.0              h69a702a_15    conda-forge
libgfortran5              11.2.0              h5c6108e_15    conda-forge
libgomp                   11.2.0              h1d223b6_15    conda-forge
liblapack                 3.9.0           14_linux64_openblas    conda-forge
liblapacke                3.9.0           14_linux64_openblas    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.20          pthreads_h78a6416_0    conda-forge
libprotobuf               3.20.0               h6239696_0    conda-forge
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libstdcxx-ng              11.2.0              he4da1e4_15    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libzlib                   1.2.11            h166bdaf_1014    conda-forge
logmuse                   0.2.6              pyh8c360ce_0    conda-forge
markupsafe                2.1.1            py39hb9d737c_1    conda-forge
multidict                 6.0.2            py39hb9d737c_1    conda-forge
nbformat                  5.3.0              pyhd8ed1ab_0    conda-forge
ncurses                   6.3                  h27087fc_1    conda-forge
numpy                     1.22.3           py39h18676bf_2    conda-forge
oauth2client              4.1.3                      py_0    conda-forge
openssl                   1.1.1n               h166bdaf_0    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pandas                    1.4.2            py39h1832856_1    conda-forge
paramiko                  2.10.3             pyhd8ed1ab_0    conda-forge
peppy                     0.31.2             pyhd8ed1ab_2    conda-forge
pip                       22.0.4             pyhd8ed1ab_0    conda-forge
plac                      1.3.5              pyhd8ed1ab_0    conda-forge
pluggy                    1.0.0            py39hf3d152e_3    conda-forge
ply                       3.11                       py_1    conda-forge
prettytable               3.2.0              pyhd8ed1ab_0    conda-forge
protobuf                  3.20.0           py39h5a03fae_4    conda-forge
psutil                    5.9.0            py39hb9d737c_1    conda-forge
pulp                      2.6.0            py39hf3d152e_1    conda-forge
py                        1.11.0             pyh6c4a22f_0    conda-forge
pyasn1                    0.4.8                      py_0    conda-forge
pyasn1-modules            0.2.7                      py_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pygments                  2.11.2             pyhd8ed1ab_0    conda-forge
pynacl                    1.5.0            py39hb9d737c_1    conda-forge
pyopenssl                 22.0.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.8              pyhd8ed1ab_0    conda-forge
pyrsistent                0.18.1           py39hb9d737c_1    conda-forge
pysftp                    0.2.9                      py_1    conda-forge
pysocks                   1.7.1            py39hf3d152e_5    conda-forge
pytest                    7.1.1            py39hf3d152e_1    conda-forge
python                    3.9.12          h9a8a25e_1_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-fastjsonschema     2.15.3             pyhd8ed1ab_0    conda-forge
python-irodsclient        1.1.3              pyhd8ed1ab_0    conda-forge
python_abi                3.9                      2_cp39    conda-forge
pytz                      2022.1             pyhd8ed1ab_0    conda-forge
pyu2f                     0.1.5              pyhd8ed1ab_0    conda-forge
pyyaml                    6.0              py39hb9d737c_4    conda-forge
ratelimiter               1.2.0                   py_1002    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
requests                  2.27.1             pyhd8ed1ab_0    conda-forge
retry                     0.9.2                      py_0    conda-forge
rsa                       4.8                pyhd8ed1ab_0    conda-forge
s3transfer                0.5.2              pyhd8ed1ab_0    conda-forge
setuptools                62.1.0           py39hf3d152e_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
slacker                   0.14.0                     py_0    conda-forge
smart_open                5.2.1              pyhd8ed1ab_0    conda-forge
smmap                     3.0.5              pyh44b312d_0    conda-forge
snakemake                 7.3.8                hdfd78af_0    bioconda
snakemake-minimal         7.3.8              pyhdfd78af_0    bioconda
sqlite                    3.38.2               h4ff8645_0    conda-forge
stone                     3.3.1              pyhd8ed1ab_0    conda-forge
stopit                    1.1.2                      py_0    conda-forge
tabulate                  0.8.9              pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
toposort                  1.7                pyhd8ed1ab_0    conda-forge
traitlets                 5.1.1              pyhd8ed1ab_0    conda-forge
typing-extensions         4.1.1                hd8ed1ab_0    conda-forge
typing_extensions         4.1.1              pyha770c72_0    conda-forge
tzdata                    2022a                h191b570_0    conda-forge
ubiquerg                  0.6.1              pyh9f0ad1d_0    conda-forge
uritemplate               4.1.1              pyhd8ed1ab_0    conda-forge
urllib3                   1.26.9             pyhd8ed1ab_0    conda-forge
veracitools               0.1.3                      py_0    conda-forge
wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
wrapt                     1.14.0           py39hb9d737c_1    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
yarl                      1.7.2            py39hb9d737c_2    conda-forge
yte                       1.2.2            py39hf3d152e_0    conda-forge
zipp                      3.8.0              pyhd8ed1ab_0    conda-forge
zlib                      1.2.11            h166bdaf_1014    conda-forge

Any thoughts about why the memory limit in the Cookiecutter profile isn't being used by my rule? Thanks very much.

[leoisl]

Thanks for the bug report. I tested running your rule without the resources directive:

rule make_file:
    output:
        "new_file.txt"
    shell:
        "touch {output}"

and this is the job script the LSF profile gets:

#!/bin/sh
# properties = {"type": "single", "rule": "make_file", "local": false, "input": [], "output": ["new_file.txt"], "wildcards": {}, "params": {}, "log": [], "threads": 1, "resources": {"mem_mb": 1000, "disk_mb": 1000, "tmpdir": "/hps/nobackup/research/zi/leandro/temp"}, "jobid": 1, "cluster": {}}
cd /hps/nobackup/research/zi/leandro/snakemake_test && /hps/nobackup/research/zi/leandro/miniconda3/bin/python -m snakemake --snakefile '/hps/nobackup/research/zi/leandro/snakemake_test/Snakefile' 'new_file.txt' --allowed-rules 'make_file' --cores 'all' --attempt 1 --force-use-threads  --wait-for-files '/hps/nobackup/research/zi/leandro/snakemake_test/.snakemake/tmp.y4ulcv9w' --force --keep-target-files --keep-remote --max-inventory-time 0 --nocolor --notemp --no-hooks --nolock --ignore-incomplete --skip-script-cleanup  --conda-frontend 'mamba' --wrapper-prefix 'https://github.com/snakemake/snakemake-wrappers/raw/' --latency-wait 10 --scheduler 'greedy' --scheduler-solver-path '/hps/nobackup/research/zi/leandro/miniconda3/bin' --default-resources 'mem_mb=max(2*input.size_mb, 1000)' 'disk_mb=max(2*input.size_mb, 1000)' 'tmpdir=system_tmpdir' --mode 2 && exit 0 || exit 1

Even though we have not specified a mem_mb for this rule, snakemake tells us that this is the resources for this rule in the properties of the job script: "resources": {"mem_mb": 1000, "disk_mb": 1000, "tmpdir": "/hps/nobackup/research/zi/leandro/temp"}, and this has to be prioritised over the profile's default memory, because usually the user does specify mem_mb. I think this is the same issue as the default threads: snakemake has defaults for mem_mb if the user does not specify it, and there is no way for us to differentiate between the user specifying that a rule needs 1000 MB of RAM or not specifying the mem_mb directive (and snakemake implicitly putting 1000 MB of RAM as default). In other words, I think the solution for this is to actually remove default_mem_mb from the cookie-cutter, and let snakemake controls the default mem_mb.

PS: the job script snakemake produces has the following default resources line: --default-resources 'mem_mb=max(2*input.size_mb, 1000)' 'disk_mb=max(2*input.size_mb, 1000)' 'tmpdir=system_tmpdir'. We could potentially change the job script so that the default mem_mb matches LSF profile default mem_mb, but this would not be enough as we would still need to know if the user explicitly asked for some specific mem_mb or not, in order to know how much memory to request when bsubing the job.

[mbhall88]

I believe new issue mentioned above is a duplicate of #41? Please raise in a new issue if not.