AttributeError: 'OptLM' object has no attribute 'weight_home'

[pxc3113]

(textgen) C:\Users\35934>python -m flexgen.flex_opt --model facebook/opt-1.3b
Downloading (…)okenizer_config.json: 100%|████████████████████████████████████████████████████| 685/685 [00:00<?, ?B/s]
C:\Users\35934\miniconda3\envs\textgen\lib\site-packages\huggingface_hub\file_download.py:133: UserWarning: huggingface_hub cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in C:\Users\35934.cache\huggingface\hub. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the HF_HUB_DISABLE_SYMLINKS_WARNING environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations.
To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to see activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development
warnings.warn(message)
Downloading (…)lve/main/config.json: 100%|████████████████████████████████████████████████████| 651/651 [00:00<?, ?B/s]
Downloading (…)olve/main/vocab.json: 100%|███████████████████████████████████████████| 899k/899k [00:01<00:00, 808kB/s]
Downloading (…)olve/main/merges.txt: 100%|███████████████████████████████████████████| 456k/456k [00:00<00:00, 571kB/s]
Downloading (…)cial_tokens_map.json: 100%|████████████████████████████████████████████████████| 221/221 [00:00<?, ?B/s]
Exception in thread Thread-2 (copy_worker_func):
Traceback (most recent call last):
Exception in thread Thread-3 (copy_worker_func):
Traceback (most recent call last):
File "C:\Users\35934\miniconda3\envs\textgen\lib\threading.py", line 1016, in _bootstrap_inner
File "C:\Users\35934\miniconda3\envs\textgen\lib\threading.py", line 1016, in _bootstrap_inner
model size: 2.443 GB, cache size: 0.398 GB, hidden size (prefill): 0.008 GB
init weight...
Exception in thread Thread-4 (copy_worker_func):
Exception in thread Thread-5 (copy_worker_func):
Traceback (most recent call last):
Traceback (most recent call last):
File "C:\Users\35934\miniconda3\envs\textgen\lib\threading.py", line 1016, in _bootstrap_inner
File "C:\Users\35934\miniconda3\envs\textgen\lib\threading.py", line 1016, in _bootstrap_inner
Traceback (most recent call last):
self.run()
File "C:\Users\35934\miniconda3\envs\textgen\lib\threading.py", line 953, in run
File "C:\Users\35934\miniconda3\envs\textgen\lib\runpy.py", line 196, in _run_module_as_main
self.run()
File "C:\Users\35934\miniconda3\envs\textgen\lib\threading.py", line 953, in run
self.run()
File "C:\Users\35934\miniconda3\envs\textgen\lib\threading.py", line 953, in run
self.run()
File "C:\Users\35934\miniconda3\envs\textgen\lib\threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
self._target(*self._args, **self._kwargs)
File "C:\Users\35934\miniconda3\envs\textgen\lib\site-packages\flexgen\pytorch_backend.py", line 880, in copy_worker_func
File "C:\Users\35934\miniconda3\envs\textgen\lib\site-packages\flexgen\pytorch_backend.py", line 880, in copy_worker_func
self._target(*self._args, **self._kwargs)
self._target(*self.args, **self.kwargs)
File "C:\Users\35934\miniconda3\envs\textgen\lib\site-packages\flexgen\pytorch_backend.py", line 880, in copy_worker_func
File "C:\Users\35934\miniconda3\envs\textgen\lib\site-packages\flexgen\pytorch_backend.py", line 880, in copy_worker_func
return run_code(code, main_globals, None,
File "C:\Users\35934\miniconda3\envs\textgen\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "C:\Users\35934\miniconda3\envs\textgen\lib\site-packages\flexgen\flex_opt.py", line 1323, in
torch.cuda.set_device(cuda_id)
File "C:\Users\35934\miniconda3\envs\textgen\lib\site-packages\torch\cuda_init.py", line 350, in set_device
torch.cuda.set_device(cuda_id)
torch.cuda.set_device(cuda_id)
File "C:\Users\35934\miniconda3\envs\textgen\lib\site-packages\torch\cuda_init.py", line 350, in set_device
File "C:\Users\35934\miniconda3\envs\textgen\lib\site-packages\torch\cuda_init.py", line 350, in set_device
torch.cuda.set_device(cuda_id)
run_flexgen(args)
File "C:\Users\35934\miniconda3\envs\textgen\lib\site-packages\flexgen\flex_opt.py", line 1215, in run_flexgen
File "C:\Users\35934\miniconda3\envs\textgen\lib\site-packages\torch\cuda_init.py", line 350, in set_device
torch._C._cuda_setDevice(device)
torch._C._cuda_setDevice(device)
torch._C._cuda_setDevice(device)
AttributeError: module 'torch._C' has no attribute '_cuda_setDevice'
AttributeError: module 'torch._C' has no attribute '_cuda_setDevice'
AttributeError: module 'torch._C' has no attribute '_cuda_setDevice'
torch._C._cuda_setDevice(device)
AttributeError: module 'torch._C' has no attribute '_cuda_setDevice'
model = OptLM(opt_config, env, args.path, policy)
File "C:\Users\35934\miniconda3\envs\textgen\lib\site-packages\flexgen\flex_opt.py", line 618, in init
self.load_weight_stream = torch.cuda.Stream()
File "C:\Users\35934\miniconda3\envs\textgen\lib\site-packages\torch\cuda\streams.py", line 34, in new
return super(Stream, cls).new(cls, priority=priority, **kwargs)
File "C:\Users\35934\miniconda3\envs\textgen\lib\site-packages\torch\cuda_utils.py", line 46, in err_fn
raise RuntimeError(
RuntimeError: Tried to instantiate dummy base class Stream
Exception ignored in: <function OptLM.del at 0x00000281748FEDD0>
Traceback (most recent call last):
File "C:\Users\35934\miniconda3\envs\textgen\lib\site-packages\flexgen\flex_opt.py", line 1149, in del
self.delete_all_weights()
File "C:\Users\35934\miniconda3\envs\textgen\lib\site-packages\flexgen\flex_opt.py", line 804, in delete_all_weights
self.delete_weight(j, 0)
File "C:\Users\35934\miniconda3\envs\textgen\lib\site-packages\flexgen\flex_opt.py", line 670, in delete_weight
for x in self.weight_home[j].pop():
AttributeError: 'OptLM' object has no attribute 'weight_home'

[SeekPoint]

me too

[XiaomingXu1995]

The key Error is:
AttributeError: module 'torch._C' has no attribute '_cuda_setDevice'

you can test the environment by
import torch print(torch.cuda.is_available()), if False, run the pip install torch torchvision torchaudio to reinstall the packages.