Fix HIP multi-GPU bug

[BohanZhang0908]

Summary
To fix a problem about multi-GPU parallelism on AMD.
Modification
5 additions on the code
Others

[brucefan1983]

@Dankomaister
@jesperbygg

HIP multi-GPU bug fixed!

[brucefan1983]

@elindgren

[jesperbygg]

Excellent!

[elindgren]

Nice! Was the problem that there were still ongoing calculations on the GPU from previous compute calls? Or why the need for gpuDeviceSynchronize

[brucefan1983]

perhaps CUDA version is just lucky. Need a sync rigorously speaking.

[Dankomaister]

This is great! I will do some benchmarking :)

Btw @brucefan1983 would it be possible to lower the system size requirement for running on multiple GPUs? or is there some technical limitation for this (apart from efficiency).

Now we require that the shortest lattice constant is more than 5 times the cutoff per GPU. "The longest direction has less than 5 times of the NEP cutoff per GPU."

Since we have 8 MI250x GPUs on one node (4 physical cards) that means we need 5 x 8 x cutoff which is a huge system especially if one has a cubic box.

[brucefan1983]

This is great! I will do some benchmarking :)

Btw @brucefan1983 would it be possible to lower the system size requirement for running on multiple GPUs? or is there some technical limitation for this (apart from efficiency).

Now we require that the shortest lattice constant is more than 5 times the cutoff per GPU. "The longest direction has less than 5 times of the NEP cutoff per GPU."

Since we have 8 MI250x GPUs on one node (4 physical cards) that means we need 5 x 8 x cutoff which is a huge system especially if one has a cubic box.

Yes this requirement is based on efficiency. Perhaps it can be reduced to a minimal of 3 cutoffs. You can try to change the code to

 if (num_bins_longitudinal < 6) {
    printf("The longest direction has less than 3 times of the NEP cutoff per GPU.\n");
    printf("Please reduce the number of GPUs or increase the simulation cell size.\n");
    exit(1);
  }

But I encourage to compare the performance with increasing number of GPUs and then choose wisely.