[WSL-2] lava-dl installation: BUILD FAILED - RuntimeError: CUDA error: no kernel image is available for execution on the device #20
Comments
|
Hi @GoHeFa, it looks like your PyTorch installation does not support RTX3090 (cuda compute capability 8.6, i.e. sm 86). This is from your error message I am assuming you installed the latest PyTorch version. The solution is to build your PyTorch locally. Before you do that, let's confirm the following things
import torch
x = torch.rand(100)
x = x.to(torch.device('cuda')
python -c "import torch; print(torch.__version__)"
nvidia-smi | grep CUDA
nvcc --version | grep release
gcc --version
|
|
@bamsumit First of all, thanks for your response. After successfully installing the latest PyTorch version via: Now to your tests:
import torch
x = torch.rand(100)
x = x.to(torch.device('cuda'))Works fine without any errors with the latest PyTorch version installed (with the pip3 command depicted above). While with
(base) user@machine:~/miniconda3/lava-dl$ python -c "import torch; print(torch.__version__)"
1.10.0+cu113
(base) user@machine:~/miniconda3/lava-dl$ nvidia-smi | grep CUDA
| NVIDIA-SMI 510.00 Driver Version: 510.06 CUDA Version: 11.6 |
(base) user@machine:~/miniconda3/lava-dl$ nvcc --version | grep release
Cuda compilation tools, release 11.5, V11.5.119
(base) user@machine:~/miniconda3/lava-dl$ gcc --version
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
To verify that my CUDA installation is working I only did: (Honestly, hence describing it as "working" is not appropriate.) nvidia-smi | grep CUDA
nvcc --version | grep releaseAlso, entering |
|
So are you able to run lava-dl with |
|
Unfortunately I am noch able to build lava-dl with (base) user@machine:~/miniconda3/lava-dl$ pip3 install torch==1.10.0+cu113 torchvision==0.11.1+cu113 torchaudio==0.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html
Looking in links: https://download.pytorch.org/whl/cu113/torch_stable.html
Requirement already satisfied: torch==1.10.0+cu113 in /home/user/miniconda3/lib/python3.8/site-packages (1.10.0+cu113)
Requirement already satisfied: torchvision==0.11.1+cu113 in /home/user/miniconda3/lib/python3.8/site-packages (0.11.1+cu113)
Requirement already satisfied: torchaudio==0.10.0+cu113 in /home/user/miniconda3/lib/python3.8/site-packages (0.10.0+cu113)
Requirement already satisfied: typing-extensions in /home/user/miniconda3/lib/python3.8/site-packages (from torch==1.10.0+cu113) (4.0.0)
Requirement already satisfied: pillow!=8.3.0,>=5.3.0 in /home/user/miniconda3/lib/python3.8/site-packages (from torchvision==0.11.1+cu113) (8.4.0)
Requirement already satisfied: numpy in /home/user/miniconda3/lib/python3.8/site-packages (from torchvision==0.11.1+cu113) (1.21.2)
(base) user@machine:~/miniconda3/lava-dl$ pyb -E unit
PyBuilder version 0.13.3
Build started at 2021-12-08 18:13:13
------------------------------------------------------------
[INFO] Installing or updating plugin "pypi:pybuilder_bandit, module name 'pybuilder_bandit'"
[INFO] Processing plugin packages 'pybuilder_bandit' to be installed with {}
[INFO] Activated environments: unit
[INFO] Building lava-dl version 0.1.1
[INFO] Executing build in /home/user/miniconda3/lava-dl
[INFO] Going to execute tasks: analyze, publish
[INFO] Processing plugin packages 'flake8~=3.7' to be installed with {'upgrade': True}
[INFO] Processing plugin packages 'pypandoc~=1.4' to be installed with {'upgrade': True}
[INFO] Processing plugin packages 'setuptools>=38.6.0' to be installed with {'upgrade': True}
[INFO] Processing plugin packages 'sphinx_rtd_theme' to be installed with {}
[INFO] Processing plugin packages 'sphinx_tabs' to be installed with {}
[INFO] Processing plugin packages 'twine>=1.15.0' to be installed with {'upgrade': True}
[INFO] Processing plugin packages 'unittest-xml-reporting~=3.0.4' to be installed with {'upgrade': True}
[INFO] Processing plugin packages 'wheel>=0.34.0' to be installed with {'upgrade': True}
[INFO] Creating target 'build' VEnv in '/home/user/miniconda3/lava-dl/target/venv/build/cpython-3.8.12.final.0'
[INFO] Processing dependency packages 'requirements.txt' to be installed with {}
[INFO] Creating target 'test' VEnv in '/home/user/miniconda3/lava-dl/target/venv/test/cpython-3.8.12.final.0'
[INFO] Processing dependency packages 'requirements.txt' to be installed with {}
[INFO] Executing flake8 on project sources.
[INFO] Running unit tests
[INFO] Executing unit tests from Python modules in /home/user/miniconda3/lava-dl/tests/lava
/home/user/miniconda3/lava-dl/target/venv/build/cpython-3.8.12.final.0/lib/python3.8/site-packages/torch/cuda/__init__.py:104: UserWarning:
NVIDIA GeForce RTX 3090 with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the NVIDIA GeForce RTX 3090 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/
warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
------------------------------------------------------------
BUILD FAILED - RuntimeError: CUDA error: no kernel image is available for execution on the device (/home/user/miniconda3/lava-dl/tests/lava/lib/dl/slayer/neuron/test_adrf_iz.py:45)
------------------------------------------------------------
Build finished at 2021-12-08 18:13:19
Build took 6 seconds (6421 ms) |
|
I would try re-installing nvidia driver and nvcc compiler for |
bamsumit
changed the title
|
@mgkwill Unfortunately, it's currently not possible for me to test the suggestion, hence I can't tell you. |
|
Has anyone else had this issue? I'm having the exact same problem at the moment, here are some of the requested outputs:
|
|
@VishalPathak-GTRI are you also running it in WSL-2? First thing I would check is to check if you can compile and run cuda code using nvcc. |
|
Hello everyone, I'm able to train pytorch model on gpu using cuda however with the same set up on LAVA blocks gives an error. nvcc: NVIDIA (R) Cuda compiler driver NVIDIA-SMI 495.29.05 Driver Version: 495.29.05 CUDA Version: 11.5 error : |
|
@uslumt , can you try updating your nvcc compiler to v 11.5? |
bamsumit
added
0-wontfix
I tried to build lava-dl on my local WSL2 (with Ubuntu 20.04) machine with an RTX 3090 (after successfully installing lava and lava-optimization) and followed therefor the given installation instructions. (I already had a working CUDA 11.5 installation on this machine.)
Unfortunately I received this error:
Is the RTX 3090 currently not supported? How to resolve this issue? Are there otherwise any workarounds?
Here is the full output:
I afterwards installed the latest PyTorch version directly from the PyTorch page and re-ran the building process but ran into a similar issue:
The text was updated successfully, but these errors were encountered: