RuntimeError: CUDA error: unknown error

[Shaosifan]

I tried to use one 4090 to train this model, but I met the following problem. What may cause this, thank you!

File "/mnt/c/Codes/referring-segmentation/work-2-SAM-guided/LISA-main/train_ds.py", line 477, in train model.step() File "/home/ls/anaconda3/envs/SAM-RIS-py310/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 2213, in step self._take_model_step(lr_kwargs) File "/home/ls/anaconda3/envs/SAM-RIS-py310/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 2119, in _take_model_step self.optimizer.step() File "/home/ls/anaconda3/envs/SAM-RIS-py310/lib/python3.10/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 1910, in step self._optimizer_step(i) File "/home/ls/anaconda3/envs/SAM-RIS-py310/lib/python3.10/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 1816, in _optimizer_step self.optimizer.step() File "/home/ls/anaconda3/envs/SAM-RIS-py310/lib/python3.10/site-packages/torch/optim/optimizer.py", line 140, in wrapper out = func(*args, **kwargs) File "/home/ls/anaconda3/envs/SAM-RIS-py310/lib/python3.10/site-packages/deepspeed/ops/adam/fused_adam.py", line 157, in step state['exp_avg_sq'] = torch.zeros_like(p.data) RuntimeError: CUDA error: unknown error