v4-CMake

[otbrown]

Hi @TysonRayJones,

Creating a PR to discuss and critique my work on the new build system :) I'll use this as a place to keep notes and thoughts too.

I've been through and identified the following open issues which I'll try to address, as well as my own biases:

• install prefix and library naming conventions #351 (Didn't know you could do that through CMake, but I like it!)
• QuEST/CMakeLists.txt broken on arm #389 and Illegal instruction when AVX instruction set is unsupported #259 - I should speak to Ali about this too. I have to say my gut feeling is "turn on -O3 a.k.a. Release and forget about it" is the optimal strategy these days, but I don't normally support such a range of environments...
• Inconsistent CMake naming #154 Suspect this one is deprecated!

Notes:

• I've bumped from 3.7 to 3.18 (which happens to be the version available in Debian 11 apt) but consider it a placeholder until the build system is complete, at which point I'll try to figure out a real minimum. I'd be very surprised if it's below 3.13, but might be willing to forgo 3.2x features...
• I personally dislike the build in place user executable functionality that was in the old CMake, but I'll make sure it continues to be supported alongside more a standard library build/install target.
• I will need support testing against QuESTLink/Windows things!

[TysonRayJones]

Wew sounds good! I'm happy to cop the Windows testing. I'll setup CI too once the build is runnable (and probably a linter, since I've already slopped in some spaghetti).

I agree it's clumsy/overbearing/amateurish to build the user's executable, though I suspect a good chunk of QuEST's user base have physics backgrounds and lack the requisite skills to make their own build, or integrate QuEST into a bigger stack. I want to make compilation as easy as possible for that group somehow. I'm open to better ideas!

[otbrown]

Can confirm that I can build and run a (very simple) example code using the available new API with and without multithreading enabled (though it doesn't deploy with MT at one qubit...), so now I can get started on the fun stuff.

Notes:

• Include paths need to be rationalised but this goes hand-in-hand with setting up the install targets, since we can use separate generators for the build and install includes.
• Once I've done that I can also reimplement the user executable functionality.
• Similarly, I can now look at install prefix and library naming conventions #351.
• On the other hand I can't make much headway with the GPU build(s) until tests are fixed.

[otbrown]

Bump to CMake 3.14 incoming for policy CMP0082.

[TysonRayJones]

Ahh I'm sorry for the rebasing and conflicting the PR! I hadn't realised the source files were being modified too. I'm intending to rename some files (structures to matrices since it' already quite long; other new data structures will have their own files) - is it okay if I rebase these too and we sort out the conflicts later? 😅

Btw are the relative path includes changed in b87b7f5 essential for CMake building? I was following Kolpackov's project structure (but preserving src and include subdirecs), and enjoyed the explicit separation (e.g. like in validation.cpp) of:

// API
#include "quest/include/modes.h"
#include "quest/include/environment.h"
#include "quest/include/qureg.h"
#include "quest/include/structures.h"

// core
#include "quest/src/core/errors.hpp"
#include "quest/src/core/bitwise.hpp"
#include "quest/src/core/memory.hpp"
#include "quest/src/core/utilities.hpp"
#include "quest/src/comm/comm_config.hpp"
#include "quest/src/cpu/cpu_config.hpp"
#include "quest/src/gpu/gpu_config.hpp"

// external library
#include <iostream>
#include <cstdlib>
#include <vector>
#include <map>

It was also consistent in that all paths were relative to the directory containing quest.

If I understand right, with the new includes, some are relative to include/ (fair enough)

#include "types.h"

and the rest are relative to the source file location (e.g. this in comm/):

#include "../core/errors.hpp"

This seems less elegant and consistent - is it necessary for compilation, or offer another benefit I've missed?

[otbrown]

No problem! Any conflicts should be pretty trivial to resolve, as I wouldn't expect to touch anything more than includes, so happy to just keep merging as and when.

Apologies, I'd assumed it was just an artifact of the make build 😅

Advantages of the way I've done it:

• Install directory has a standard structure: include, libs, bins. Users just need to add -I<installdir>/include to build their programs using the public API.
• Relative includes don't require a -I flag at all at compile time, and are slightly less sensitive to changes in the project structure.

However, through CMake, all things are possible 😉

• Build and install interfaces for include directories can be set separately so I can make CMake export whatever we like as as the install include interface, and we can just tell users how to correctly write their includes relative to that.
• Setting the build includes to anything is trivial and really doesn't affect users at all 🤷‍♂️

So very happy to change it all back, it's just a find/replace job! It's your project, so let me know which you'd prefer.

[otbrown]

Library renaming is now available -- the user can either just provide a custom string to produce lib${LIB_NAME}.so, or turn on VERBOSE_LIB_NAME to append to the lib name based on configuration options. This should all play nicely with the install export stuff too, though I haven't tested that.

GPU acceleration and MPI distribution are now also available, but there are two issues:

• cuQuantum: nvidia don't provide a CMake integration, and I don't realllllly want to write a findcuQuantum for them, though I will if I have to. Need to ponder that one.
• HIP: It looks as if the HIP version builds just by using the HIP compiler to compile the CUDA which is nice, but may present some issues for CMake. The simplest solution (for me) is to just split ENABLE_GPU_ACCELERATION into ENABLE_CUDA and ENABLE_HIP, but will see if there's a more creative solution! Never mind, there's already a USE_HIP option at play in v3, so I'll just make use of that! 😁

[otbrown]

@TysonRayJones is the plan to support HIP in the same way (i.e. by setting USE_HIP)?

[otbrown]

Oh and what's the minimum required CUDA version?

[TysonRayJones]

• cuQuantum: nvidia don't provide a CMake integration, and I don't realllllly want to write a findcuQuantum for them, though I will if I have to. Need to ponder that one.

Ahh annoying - it's a little unexpected that they provide GNU make examples, but it makes me feel less out of touch :^) . Maybe they're expecting most users to build cuStateVec via CUDA Quantum (which we should expressly not do).

@TysonRayJones is the plan to support HIP in the same way (i.e. by setting USE_HIP)?

Yea! But you can choose any name that makes sense / is consistent, like COMPILE_HIP.

Note that the GPU code (even when not compiling cuStateVec) should use Thrust, because:

• it's included in the CUDA toolkit
• it avoids lots of boilerplate/custom-kernels to do reductions (e.g. compare this lovely Thrust version of this custom kernel and caller).

This requires that HIP can wrap/map the Thrust functors and calls to rocThrust, but I don't know if that's always included with ROCm. I've just been hoping and praying! In any case, feel free to defer HIP support for the moment in case it causes a problem, and we can re-enable HIP in a later patch.

Oh and what's the minimum required CUDA version?

It will be build-configuration dependant:

• if compiling cuStateVec, it will be CC >= 7 as per the cuStateVec minimum here.
• otherwise, it'll be much more permissive. Thrust itself requires CC >= 2 (says ChatGPT heh), and I don't think we're using anything specific to higher CCs. I only use streams when compiling cuStateVec, for example. I do want to experiment with CUDA's occupancy API which is CC >= 6.5, but it's inessential, and we can/should use compiler-guards to support lower CC / older GPUs.

QuEST v3's default CC was 3.0, so let's just use that as a place-holder when not compiling cuStateVec ¯\(ツ)/¯

[otbrown]

most users to build cuStateVec via CUDA Quantum (which we should expressly not do)

Almost certainly! I will probably just relent and write a simple findCuQuantum. This works fine as long as standard installations of it have a consistent directory structure.

you can choose any name that makes sense / is consistent, like COMPILE_HIP

For the compiler flags COMPILE_CUDA still makes sense as what's written is CUDA whether we compile it with nvcc or hipcc, and there's no point changing the pragmas. On the other hand I may ditch ENABLE_GPU_ACCELERATION in favour of ENABLE_CUDA and ENABLE_HIP a) because I don't see a scenario where someone tries to build both, and if they do they have bigger problems than the CMake, and b) both are shorter than ENABLE_GPU_ACCELERATION. 😃

Note that the GPU code (even when not compiling cuStateVec) should use Thrust,

Good to know! Seems like both Thrust and rocThrust export CMake targets, so should at least be able to make this an explicit dependency. I'll look into it (no point adding it if it breaks builds that actually should work...).

feel free to defer HIP support

I actually looked into it already on the ARCHER2 GPU testbed, and enabling HIP support is super easy with CMake 3.21+ (for context, the latest is 3.30, so we're still well behind the curve). Shouldn't be an issue therefore +/- potential Thrust issues. I can at least get HIP support in and building for now, and then see if it actually runs in the testing/benchmarking phase.

Perfect, thanks for the versions, I'll add them in!

[otbrown]

Note: Currently #include <cuda.h> in gpu_config.cpp breaks the HIP build -- I'll include AMD support in the CMake for now and we can always just remove it later!

[otbrown]

Eurgh, so although Thrust has a CMake target, but for some reason CUDAToolkit doesn't expose it 🙄 Best option for now to make this an explicit dependency is probably to figure out what version of CUDAToolkit bundled the minimum version of Thrust and then set that as a minimum version of CUDAToolkit... We could insist on explicitly linking Thrust, but I feel it's not worth getting the user to dig through NVIDIA install directories when the code would actually link fine.