Running on macos / mps

[Chris-toe-pher]

Segment anything can be run on mps with good acceleration vs cpu. This demo fails to run when loading the deva model - "torch not compiled with cuda enabled". Is it possible to support mps?

Traceback (most recent call last):
File "/Users/chris/Documents/AI/segmentation/Tracking-Anything-with-DEVA/demo/demo_automatic.py", line 34, in
deva_model, cfg, args = get_model_and_config(parser)
File "/Users/chris/Documents/AI/segmentation/Tracking-Anything-with-DEVA/deva/inference/eval_args.py", line 65, in get_model_and_config
network = DEVA(config).cuda().eval()
File "/Users/chris/miniconda3/envs/TA-DEVA/lib/python3.9/site-packages/torch/nn/modules/module.py", line 905, in cuda
return self._apply(lambda t: t.cuda(device))
File "/Users/chris/miniconda3/envs/TA-DEVA/lib/python3.9/site-packages/torch/nn/modules/module.py", line 797, in _apply
module._apply(fn)
File "/Users/chris/miniconda3/envs/TA-DEVA/lib/python3.9/site-packages/torch/nn/modules/module.py", line 797, in _apply
module._apply(fn)
File "/Users/chris/miniconda3/envs/TA-DEVA/lib/python3.9/site-packages/torch/nn/modules/module.py", line 820, in _apply
param_applied = fn(param)
File "/Users/chris/miniconda3/envs/TA-DEVA/lib/python3.9/site-packages/torch/nn/modules/module.py", line 905, in
return self._apply(lambda t: t.cuda(device))
File "/Users/chris/miniconda3/envs/TA-DEVA/lib/python3.9/site-packages/torch/cuda/init.py", line 239, in _lazy_init
raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

[hkchengrex]

I don't have a Mac device and cannot test it but I suppose you can replace all the .cuda() and cuda.amp calls with mps equivalent.

[Chris-toe-pher]

If you don't have a Mac that is a good reason 👍 .
I tried replacing the cuda specific things, also there are some datatypes that aren't supported on mps like int64... and some functions that fallback to cpu. I got as far as it running but with an error from MPSGraphUtilities.mm scrolling by. After 10 minutes, the progress bar showed 2 images had been completed, then it quit with an error, and no output images. It looks like it is using up more than the 64GB of unified memory, I'm not sure I can fix it further without understanding each part that's being used.

[hkchengrex]

If you try the example video, i.e.,

python demo/demo_with_text.py --chunk_size 4 \
--img_path ./example/vipseg/images/12_1mWNahzcsAc \ 
--amp --temporal_setting semionline \
--size 480 \
--output ./example/output --prompt person.hat.horse

, it is quite unlikely that it uses more than a few GB of memory.

If it processed two frames without any output images, it sounds like the error occurred during consensus. You can test using the online mode (consensus disabled).

[Chris-toe-pher]

I was running the other one - demo_automatic.py
The demo_with_text has a different error from GroundingDINO, I'll have a look into that tomorrow.

[hkchengrex]

Ref: hkchengrex/Cutie#14