COIN evaluation is too slow when using single A100

[bluehawk2k]

Hi, many thanks for your great work!
I'm trying to reproduce your result by training your algorithm with COIN dataset (/scripts/coin/live1+.sh)
Training has been finished for 50 hours by using just one single A100, and it seems that the evaluation script runs automatically once the training process is finished.
However, this evaluation process is too slow (about 20 sec/iter speed, and it takes more than 100 hours) when I use one single A100.
When we run live1+.sh script with multiple A100, evaluation processing time isn't more than 1 hour.
I'd really appreciate it if you could look into this issue.

[chenjoya]

Hi, thanks for the try! Do you mean that the evaluation on single-gpu is unusually slow but multi-gpu is normal?

[bluehawk2k]

Yes, right.
When I run /scripts/coin/live1+.sh with only one single GPU, training time takes about 50 hours, but evaluation time takes too longer (about 100+ hours). Please see below image.