How to cancel a running job? 怎么停止一个任务？

[Spico197]

When opencompass is paired with slurm, how to cancel running evaluation jobs, please?

请问使用slurm集群进行评测时，如果想要停止一个任务及slurm集群中的jobs，应该怎么操作啊？

[yingfhu]

Use squeue and scancel
SlurmSequentialRunner is recommanded to better manage jobs than SlurmRunner

[Spico197]

Thank you very much for your reply~ But opencompass will restart the jobs automatically, so squeue and scancel need to be called multiple times, especially when many jobs are running.

[yingfhu]

if the main process is still on, the retry behavior will be trigger. You can set retry=0 for Runner in config.
Retry will not work If the main process is being killed.

[Spico197]

I see. Thank you very much for your immediate reply~ It'll be much better if there's a tool to manage processes and attached slurm jobs❤️.

[yingfhu]

SlurmSequentialRunner will dispatch one job at a time until there is no resources. It might be inconvenient to manage a single task when you launched a lot of parallel tasks. Use --debug to prepare your task previously is recommended.