Calculation of alignments step died

[AlexSongh]

Hi, the screenshot of the error log was below.

And my HPC configuration is:

#!/bin/bash

#SBATCH --partition=largemem
#SBATCH --nodes=1
#SBATCH --cpus-per-task=16
#SBATCH --tasks-per-node=1
#SBATCH --mem=800G
#SBATCH --time=60:00:00

source ~/.bashrc
conda activate metaeuk
cd /scratch/raskin_root/raskin0/hangsong/Ann_Arbor.UMich/contigs_includingBioFilterEffluent_EUK_classification
mkdir -p MetaEuk_results
mkdir -p tmp

EUK_PROFILES="/nfs/turbo/cee-raskin/hangsong/metaeuk_db/MMETSP_uniclust50_MERC_profiles"
metaeuk easy-predict contigs_includingBioFilterEffluent.EUK_KmerMajority_min3000bp_CAT_min1000bp.fasta "${EUK_PROFILES}" MetaEuk_results/predRedRedund_prot.fasta tmp --metaeuk-eval 0.0001 -e 100 --exhaustive-search 1 --min-ungapped-score 35 --min-length 40 --threads 16 --split-memory-limit 300G --min-exon-aa 20 --metaeuk-tcov 0.6 --local-tmp tmp --disk-space-limit 800G;

Do you know any possible reasons to cause this? Are there any solutions? Were any of the flags have problems?
Thanks!

[milot-mirdita]

Can you please post the full log output of the job?

[AlexSongh]

Yes, below is my full log. Thank you @milot-mirdita !

easy-predict contigs_includingBioFilterEffluent.EUK_KmerMajority_min3000bp_CAT_min1000bp.fasta /nfs/turbo/cee-raskin/hangsong/metaeuk_db/MMETSP_uniclust50_MERC_profiles MetaEuk_results/predRedRedund_prot.fasta tmp --metaeuk-eval 0.0001 -e 100 --exhaustive-search 1 --min-ungapped-score 35 --min-length 40 --threads 16 --split-memory-limit 300G --min-exon-aa 20 --metaeuk-tcov 0.6 --local-tmp tmp --disk-space-limit 800G

MMseqs Version: 6.a5d39d9
Substitution matrix aa:blosum62.out,nucl:nucleotide.out
Add backtrace false
Alignment mode 2
Alignment mode 0
Allow wrapped scoring false
E-value threshold 100
Seq. id. threshold 0
Min alignment length 0
Seq. id. mode 0
Alternative alignments 0
Coverage threshold 0
Coverage mode 0
Max sequence length 65535
Compositional bias 1
Compositional bias 1
Max reject 2147483647
Max accept 2147483647
Include identical seq. id. false
Preload mode 0
Pseudo count a substitution:1.100,context:1.400
Pseudo count b substitution:4.100,context:5.800
Score bias 0
Realign hits false
Realign score bias -0.2
Realign max seqs 2147483647
Correlation score weight 0
Gap open cost aa:11,nucl:5
Gap extension cost aa:1,nucl:2
Zdrop 40
Threads 16
Compressed 0
Verbosity 3
Seed substitution matrix aa:VTML80.out,nucl:nucleotide.out
Sensitivity 4
k-mer length 0
k-score seq:2147483647,prof:2147483647
Alphabet size aa:21,nucl:5
Max results per query 300
Split database 0
Split mode 2
Split memory limit 300G
Diagonal scoring true
Exact k-mer matching 0
Mask residues 1
Mask residues probability 0.9
Mask lower case residues 0
Minimum diagonal score 35
Spaced k-mers 1
Spaced k-mer pattern
Local temporary path tmp
Rescore mode 0
Remove hits by seq. id. and coverage false
Sort results 0
Mask profile 1
Profile E-value threshold 0.001
Global sequence weighting false
Allow deletions false
Filter MSA 1
Use filter only at N seqs 0
Maximum seq. id. threshold 0.9
Minimum seq. id. 0.0
Minimum score per column -20
Minimum coverage 0
Select N most diverse seqs 1000
Pseudo count mode 0
Gap pseudo count 10
Min codons in orf 40
Max codons in length 32734
Max orf gaps 2147483647
Contig start mode 2
Contig end mode 2
Orf start mode 1
Forward frames 1,2,3
Reverse frames 1,2,3
Translation table 1
Translate orf 0
Use all table starts false
Offset of numeric ids 0
Create lookup 0
Add orf stop false
Overlap between sequences 0
Sequence split mode 1
Header split mode 0
Chain overlapping alignments 0
Merge query 1
Search type 0
Search iterations 1
Start sensitivity 4
Search steps 1
Exhaustive search mode true
Filter results during exhaustive search 0
Strand selection 1
LCA search mode false
Disk space limit 800G
MPI runner
Force restart with latest tmp false
Remove temporary files false
maximal combined evalue of an optimal set 0.0001
minimal length ratio between combined optimal set and target 0.6
Maximal intron length 10000
Minimal intron length 15
Minimal exon length aa 20
Maximal overlap of exons 10
Maximal number of exon sets 1
Gap open penalty -1
Gap extend penalty -1
allow same-strand overlaps 0
translate codons to AAs 0
write target key instead of accession 0
write fragment contig coords 0
Reverse AA Fragments 0

createdb contigs_includingBioFilterEffluent.EUK_KmerMajority_min3000bp_CAT_min1000bp.fasta tmp/589055901896411739/contigs --dbtype 2 --compressed 0 -v 3

Converting sequences
[=
Time for merging to contigs_h: 0h 0m 0s 16ms
Time for merging to contigs: 0h 0m 0s 53ms
Database type: Nucleotide
Time for processing: 0h 0m 0s 542ms
Create directory tmp/589055901896411739/tmp_predict
Enforcing exhaustive profile search mode due to profile target database
predictexons tmp/589055901896411739/contigs /nfs/turbo/cee-raskin/hangsong/metaeuk_db/MMETSP_uniclust50_MERC_profiles tmp/589055901896411739/MetaEuk_calls tmp/589055901896411739/tmp_predict --sub-mat 'aa:blosum62.out,nucl:nucleotide.out' -a 0 --alignment-mode 2 --alignment-output-mode 0 --wrapped-scoring 0 -e 100 --min-seq-id 0 --min-aln-len 0 --seq-id-mode 0 --alt-ali 0 -c 0 --cov-mode 0 --max-seq-len 65535 --comp-bias-corr 1 --comp-bias-corr-scale 1 --max-rejected 2147483647 --max-accept 2147483647 --add-self-matches 0 --db-load-mode 0 --pca substitution:1.100,context:1.400 --pcb substitution:4.100,context:5.800 --score-bias 0 --realign 0 --realign-score-bias -0.2 --realign-max-seqs 2147483647 --corr-score-weight 0 --gap-open aa:11,nucl:5 --gap-extend aa:1,nucl:2 --zdrop 40 --threads 16 --compressed 0 -v 3 --seed-sub-mat 'aa:VTML80.out,nucl:nucleotide.out' -s 4 -k 0 --k-score seq:2147483647,prof:2147483647 --alph-size aa:21,nucl:5 --max-seqs 300 --split 0 --split-mode 2 --split-memory-limit 300G --diag-score 1 --exact-kmer-matching 0 --mask 1 --mask-prob 0.9 --mask-lower-case 0 --min-ungapped-score 35 --spaced-kmer-mode 1 --local-tmp tmp --rescore-mode 0 --filter-hits 0 --sort-results 0 --mask-profile 1 --e-profile 0.001 --wg 0 --allow-deletion 0 --filter-msa 1 --filter-min-enable 0 --max-seq-id 0.9 --qid '0.0' --qsc -20 --cov 0 --diff 1000 --pseudo-cnt-mode 0 --gap-pc 10 --min-length 40 --max-length 32734 --max-gaps 2147483647 --contig-start-mode 2 --contig-end-mode 2 --orf-start-mode 1 --forward-frames 1,2,3 --reverse-frames 1,2,3 --translation-table 1 --translate 0 --use-all-table-starts 0 --id-offset 0 --create-lookup 0 --add-orf-stop 0 --sequence-overlap 0 --sequence-split-mode 1 --headers-split-mode 0 --chain-alignments 0 --merge-query 1 --search-type 0 --num-iterations 1 --start-sens 4 --sens-steps 1 --exhaustive-search 1 --exhaustive-search-filter 0 --strand 1 --lca-search 0 --disk-space-limit 800G --force-reuse 0 --remove-tmp-files 0 --metaeuk-eval 0.0001 --metaeuk-tcov 0.6 --max-intron 10000 --min-intron 15 --min-exon-aa 20 --max-overlap 10 --max-exon-sets 1 --set-gap-open -1 --set-gap-extend -1 --reverse-fragments 0

extractorfs /scratch/raskin_root/raskin0/hangsong/Ann_Arbor.UMich/contigs_includingBioFilterEffluent_EUK_classification/tmp/589055901896411739/contigs /scratch/raskin_root/raskin0/hangsong/Ann_Arbor.UMich/contigs_includingBioFilterEffluent_EUK_classification/tmp/589055901896411739/tmp_predict/9835964812102503863/nucl_6f --min-length 40 --max-length 32734 --max-gaps 2147483647 --contig-start-mode 2 --contig-end-mode 2 --orf-start-mode 1 --forward-frames 1,2,3 --reverse-frames 1,2,3 --translation-table 1 --translate 0 --use-all-table-starts 0 --id-offset 0 --create-lookup 0 --threads 16 --compressed 0 -v 3

[=================================================================] 15.20K 0s 406ms
Time for merging to nucl_6f_h: 0h 0m 0s 630ms
Time for merging to nucl_6f: 0h 0m 0s 858ms
Time for processing: 0h 0m 2s 372ms
translatenucs /scratch/raskin_root/raskin0/hangsong/Ann_Arbor.UMich/contigs_includingBioFilterEffluent_EUK_classification/tmp/589055901896411739/tmp_predict/9835964812102503863/nucl_6f /scratch/raskin_root/raskin0/hangsong/Ann_Arbor.UMich/contigs_includingBioFilterEffluent_EUK_classification/tmp/589055901896411739/tmp_predict/9835964812102503863/aa_6f --translation-table 1 --add-orf-stop 0 -v 3 --compressed 0 --threads 16

[=================================================================] 470.66K 0s 958ms
Time for merging to aa_6f: 0h 0m 0s 503ms
Time for processing: 0h 0m 1s 536ms
Create directory /scratch/raskin_root/raskin0/hangsong/Ann_Arbor.UMich/contigs_includingBioFilterEffluent_EUK_classification/tmp/589055901896411739/tmp_predict/9835964812102503863/tmp_search
search /scratch/raskin_root/raskin0/hangsong/Ann_Arbor.UMich/contigs_includingBioFilterEffluent_EUK_classification/tmp/589055901896411739/tmp_predict/9835964812102503863/aa_6f /nfs/turbo/cee-raskin/hangsong/metaeuk_db/MMETSP_uniclust50_MERC_profiles /scratch/raskin_root/raskin0/hangsong/Ann_Arbor.UMich/contigs_includingBioFilterEffluent_EUK_classification/tmp/589055901896411739/tmp_predict/9835964812102503863/search_res /scratch/raskin_root/raskin0/hangsong/Ann_Arbor.UMich/contigs_includingBioFilterEffluent_EUK_classification/tmp/589055901896411739/tmp_predict/9835964812102503863/tmp_search --sub-mat 'aa:blosum62.out,nucl:nucleotide.out' -a 0 --alignment-mode 2 --alignment-output-mode 0 --wrapped-scoring 0 -e 100 --min-seq-id 0 --min-aln-len 20 --seq-id-mode 0 --alt-ali 0 -c 0 --cov-mode 0 --max-seq-len 65535 --comp-bias-corr 1 --comp-bias-corr-scale 1 --max-rejected 2147483647 --max-accept 2147483647 --add-self-matches 0 --db-load-mode 0 --pca substitution:1.100,context:1.400 --pcb substitution:4.100,context:5.800 --score-bias 0 --realign 0 --realign-score-bias -0.2 --realign-max-seqs 2147483647 --corr-score-weight 0 --gap-open aa:11,nucl:5 --gap-extend aa:1,nucl:2 --zdrop 40 --threads 16 --compressed 0 -v 3 --seed-sub-mat 'aa:VTML80.out,nucl:nucleotide.out' -s 4 -k 0 --k-score seq:2147483647,prof:2147483647 --alph-size aa:21,nucl:5 --max-seqs 300 --split 0 --split-mode 2 --split-memory-limit 300G --diag-score 1 --exact-kmer-matching 0 --mask 1 --mask-prob 0.9 --mask-lower-case 0 --min-ungapped-score 35 --spaced-kmer-mode 1 --local-tmp tmp --rescore-mode 0 --filter-hits 0 --sort-results 0 --mask-profile 1 --e-profile 0.001 --wg 0 --allow-deletion 0 --filter-msa 1 --filter-min-enable 0 --max-seq-id 0.9 --qid '0.0' --qsc -20 --cov 0 --diff 1000 --pseudo-cnt-mode 0 --gap-pc 10 --min-length 40 --max-length 32734 --max-gaps 2147483647 --contig-start-mode 2 --contig-end-mode 2 --orf-start-mode 1 --forward-frames 1,2,3 --reverse-frames 1,2,3 --translation-table 1 --translate 0 --use-all-table-starts 0 --id-offset 0 --create-lookup 0 --add-orf-stop 0 --sequence-overlap 0 --sequence-split-mode 1 --headers-split-mode 0 --chain-alignments 0 --merge-query 1 --search-type 0 --num-iterations 1 --start-sens 4 --sens-steps 1 --exhaustive-search 1 --exhaustive-search-filter 0 --strand 1 --lca-search 0 --disk-space-limit 800G --force-reuse 0 --remove-tmp-files 0

prefilter /scratch/raskin_root/raskin0/hangsong/Ann_Arbor.UMich/contigs_includingBioFilterEffluent_EUK_classification/tmp/589055901896411739/tmp_predict/9835964812102503863/tmp_search/13289217149420800099/profileDB /scratch/raskin_root/raskin0/hangsong/Ann_Arbor.UMich/contigs_includingBioFilterEffluent_EUK_classification/tmp/589055901896411739/tmp_predict/9835964812102503863/aa_6f /scratch/raskin_root/raskin0/hangsong/Ann_Arbor.UMich/contigs_includingBioFilterEffluent_EUK_classification/tmp/589055901896411739/tmp_predict/9835964812102503863/tmp_search/13289217149420800099/pref --sub-mat 'aa:blosum62.out,nucl:nucleotide.out' --seed-sub-mat 'aa:VTML80.out,nucl:nucleotide.out' -s 4 -k 0 --k-score seq:2147483647,prof:2147483647 --alph-size aa:21,nucl:5 --max-seq-len 65535 --max-seqs 470660 --split 0 --split-mode 2 --split-memory-limit 300G -c 0 --cov-mode 0 --comp-bias-corr 1 --comp-bias-corr-scale 1 --diag-score 1 --exact-kmer-matching 0 --mask 1 --mask-prob 0.9 --mask-lower-case 0 --min-ungapped-score 35 --add-self-matches 0 --spaced-kmer-mode 1 --db-load-mode 0 --pca substitution:1.100,context:1.400 --pcb substitution:4.100,context:5.800 --local-tmp tmp --threads 16 --compressed 0 -v 3

Query database size: 73003 type: Profile
Estimated memory consumption: 1G
Target database size: 470660 type: Aminoacid
Index table k-mer threshold: 0 at k-mer size 6
Index table: counting k-mers
[=================================================================] 470.66K 0s 931ms
Index table: Masked residues: 7546309
Index table: fill
[=================================================================] 470.66K 0s 608ms
Index statistics
Entries: 40256141
DB size: 718 MB
Avg k-mer size: 0.629002
Top 10 k-mers
RRRRRR 2160
RRGRRR 1328
GRRRRR 1311
PRRRRR 1108
RPRRRR 1050
ARRRRR 1038
RGGRRR 949
GRGRRR 860
GGRRRR 780
ARGRRR 691
Time for index table init: 0h 0m 1s 993ms
Process prefiltering step 1 of 1

k-mer similarity threshold: 109
Starting prefiltering scores calculation (step 1 of 1)
Query db start 1 to 73003
Target db start 1 to 470660
[=================================================================] 73.00K 6h 51m 49s 207ms

573653.309786 k-mers per position
63152310 DB matches per sequence
25122 overflows
0 queries produce too many hits (truncated result)
36273 sequences passed prefiltering per query sequence
0 median result list length
45110 sequences with 0 size result lists
Time for merging to pref: 0h 0m 0s 128ms
Time for processing: 6h 55m 11s 449ms
result2stats /scratch/raskin_root/raskin0/hangsong/Ann_Arbor.UMich/contigs_includingBioFilterEffluent_EUK_classification/tmp/589055901896411739/tmp_predict/9835964812102503863/tmp_search/13289217149420800099/profileDB /scratch/raskin_root/raskin0/hangsong/Ann_Arbor.UMich/contigs_includingBioFilterEffluent_EUK_classification/tmp/589055901896411739/tmp_predict/9835964812102503863/aa_6f /scratch/raskin_root/raskin0/hangsong/Ann_Arbor.UMich/contigs_includingBioFilterEffluent_EUK_classification/tmp/589055901896411739/tmp_predict/9835964812102503863/tmp_search/13289217149420800099/pref /scratch/raskin_root/raskin0/hangsong/Ann_Arbor.UMich/contigs_includingBioFilterEffluent_EUK_classification/tmp/589055901896411739/tmp_predict/9835964812102503863/tmp_search/13289217149420800099/pref_count.tsv --stat linecount --tsv --threads 16 --compressed 0 -v 3

[=================================================================] 73.00K 5m 25s 242ms
Time for merging to pref_count.tsv: 0h 0m 0s 160ms
Time for processing: 0h 5m 27s 14ms
Score of forward/backward SW differ: 509 508. Q: 6796 T: 391823.
Start: Q: 3, T: 8. End: Q: 164, T 165
align /scratch/raskin_root/raskin0/hangsong/Ann_Arbor.UMich/contigs_includingBioFilterEffluent_EUK_classification/tmp/589055901896411739/tmp_predict/9835964812102503863/tmp_search/13289217149420800099/profileDB /scratch/raskin_root/raskin0/hangsong/Ann_Arbor.UMich/contigs_includingBioFilterEffluent_EUK_classification/tmp/589055901896411739/tmp_predict/9835964812102503863/aa_6f /scratch/raskin_root/raskin0/hangsong/Ann_Arbor.UMich/contigs_includingBioFilterEffluent_EUK_classification/tmp/589055901896411739/tmp_predict/9835964812102503863/tmp_search/13289217149420800099/pref /scratch/raskin_root/raskin0/hangsong/Ann_Arbor.UMich/contigs_includingBioFilterEffluent_EUK_classification/tmp/589055901896411739/tmp_predict/9835964812102503863/tmp_search/13289217149420800099/aln --sub-mat 'aa:blosum62.out,nucl:nucleotide.out' -a 0 --alignment-mode 2 --alignment-output-mode 1 --wrapped-scoring 0 -e 0.534933 --min-seq-id 0 --min-aln-len 20 --seq-id-mode 0 --alt-ali 0 -c 0 --cov-mode 0 --max-seq-len 65535 --comp-bias-corr 1 --comp-bias-corr-scale 1 --max-rejected 2147483647 --max-accept 2147483647 --add-self-matches 0 --db-load-mode 0 --pca substitution:1.100,context:1.400 --pcb substitution:4.100,context:5.800 --score-bias 0 --realign 0 --realign-score-bias -0.2 --realign-max-seqs 2147483647 --corr-score-weight 0 --gap-open aa:11,nucl:5 --gap-extend aa:1,nucl:2 --zdrop 40 --threads 16 --compressed 0 -v 3

Compute score and coverage
Query database size: 73003 type: Profile
Target database size: 470660 type: Aminoacid
Calculation of alignments
[Score of forward/backward SW differ: 258 256. Q: 28448 T: 243727.
Start: Q: 3, T: 0. End: Q: 90, T 102
Score of forward/backward SW differ: 2522 2530. Q: 6797 T: 61775.
Start: Q: 15, T: 0. End: Q: 214, T 191
Error: align died
Error: search step died
Error: predictexons step died

[Tamtatatam]

...I'll join this thread, since I came across exactly this problem today :). I fear it might be about the db size. A smaler db worked without problems for me. Would love to hear if it is possible to get around this problem.

[AlexSongh]

...I'll join this thread, since I came across exactly this problem today :). I fear it might be about the db size. A smaler db worked without problems for me. Would love to hear if it is possible to get around this problem.

Thank you so much for your reply @Tamtatatam ! May I ask what protein db did you use? Or did you do any manipulation of the original protein db profile? Thank you!

[Tamtatatam]

I used EggNOG for the run that failed, and the EggNOG db was created using mmseq with mmseqs databases eggNOG /my/db/path/EggNOG tmp and mmseqs createdb /my/db/path/EggNOG targetDB.

For the test runs that worked without any issues I used smaller custom made reference datasets (subsets of NR) and ran MetEuk either directly on the fasta or after first creating a database. That's why I assumed it is about the size, but maybe it is about how the db is created?
`

[AlexSongh]

@Tamtatatam Thank you for your reply！ How big are your subsets of NR? I used the protein profile from this link (https://wwwuser.gwdg.de/~compbiol/metaeuk/2019_11/), and it is 295 G.

How did you make the smaller NR subset? If it's a smaller db, will some of the results be compromised since they won't be found and aligned? Thank you!

[milot-mirdita]

This is not supposed to happen. Can you downgrade to the previous metaeuk version, while we investigate whats going on? The previous one should work correctly with the profile database we provide.

[AlexSongh]

This is not supposed to happen. Can you downgrade to the previous metaeuk version, while we investigate whats going on? The previous one should work correctly with the profile database we provide.

I'll try that and let you know how it goes @milot-mirdita, thank you!

[AlexSongh]

@milot-mirdita The previous version works! Thanks for the help!

[gjordaopiedade]

Hi,
I see this issue is still open. I seem to be having the exact same issue (using MMETSP_uniclust50_MERC_profiles). I have tried both installing with conda and AVX2 build.
When it gets to the alignment it runs out of memory (I have 1T available).

Compute score and coverage
Query database size: 1829 type: Profile
Target database size: 92457309 type: Aminoacid
Calculation of alignments
[=======Error: align died
Error: search step died
Error: predictexons step died

Do you think I should try an older version? Where can I find it?

Thank you!!