NVIDIA CUDA Ampere Support for Legion

[UmashankarTriforce]

Hi team!

I have compiled legion as per the instructions and have added --cuda during compilation for CUDA support.
But while running the CUDA tests, all the tests in the category run_pass fails with PTX .version 3.2 does not support .target sm_80.
I just wanted to clarify, if this is error is being raised because legion currently does not support Ampere architecture? If that's the case, are there any plans for supporting sm_80?

[streichler]

Can you give the full command line you're using for install.py? (I'm trying to figure out if you're using the cmake build or the older Makefile-based one.)

Can you try building with GPU_ARCH=80 in your environment? I don't immediately see a way to set this on the install.py command line.

[UmashankarTriforce]

I'm using ./install.py --cuda --debug to install. As per your direction, I re-built the module by using export GPU_ARCH=80, but again I'm getting the same error.

[UmashankarTriforce]

I worked on fixing this issue for a while, and the issue that I have is quite similar to the one posted here Bad PTX ISA version when using LLVM 6 or greater. I tried multiple clang and llvm versions but nothing seems to help.

[elliottslaughter]

@UmashankarTriforce if that were the issue, you ought to be able to check by compiling against LLVM 3.8.

We've been trying to replicate on our end, but Ampere GPUs are not super common yet, so we're still in the process of doing so.

[UmashankarTriforce]

@elliottslaughter I did try compiling with LLVM 3.8, but my CPU is based on AMD zen2 architecture and when I compile it, the CPU doesn't get detected correctly by LLVM. It straight defaults to generic and throws an error. With a little bit of googling, I found out that LLVM 4.0 introduces support for zen1, which means I need to be on a higher version of LLVM to have support for zen2.

[elliottslaughter]

Ok. At this point we may need someone on our team to try out Ampere because Terra doesn't support LLVM 4.0 and all versions greater than or equal to 5.0 use the NVPTX interface, which causes the issue linked above.

[pmccormick]

I'll take a shot at this... stay tuned...

[UmashankarTriforce]

@elliottslaughter @pmccormick Thank you for helping me with this! Please do let me know if something comes up :D

[streichler]

@UmashankarTriforce are you still stuck on this? I tried using the clang/llvm 11 that comes with ubuntu 20.04 and things seems to work fine, so I think you can try either a really old or a really new version of clang/llvm and have some expectation of success.

[elliottslaughter]

@UmashankarTriforce I recently built Regent on an A100 with LLVM 11, and it worked for me. I'm going to close this for now because I believe it's working, but if you run into any further issues, please feel free to reopen this issue or open a new one.