Challenge 11 - Think small, move fast

[jwagemann]

Challenge 11 - Think small, move fast

Stream 1 - Software development for weather, climate and atmosphere

Goal

Reduce the data volume exchanged between compute elements without loss of information to speedup processing, reducing energy consumption, while respecting the physics.

Mentors and skills

• Mentors: Thomas Geenen, Willem Deconinck
• Skills required:
  • Good understanding of compression algorithms and methods
  • Being able to implement solutions in C or C++
  • Some knowledge of HPC architectures and its bottlenecks
  • Familiarity of HPC ecosystems and development stacks

Note: Challenge is funded by Copernicus. Only nationals from European Union (EU) Member States and countries associated with EU’s Space Programme (currently Iceland and Norway) are eligible to participate (see Terms and Conditions).

---

Challenge description

Why do we need a solution
Moving data between computing engines in numerical (weather) codes is often the bottleneck in speeding up computations. It has also become one of the most energy consuming operations in HPC in general. Therefore, the reduction of the amount of data exchanged while at the same time preserving the information is a valuable optimisation approach both for performance as well as energy efficiency.

What could be the solution
Initial tests on systems that resemble the pre-exascale machines that are installed in Europe now, showed that indeed for numerical weather codes data movement is a bottleneck and that substantial speedups can be achieved by reducing the data volumes between CPU and GPU but also for MPI communications between GPUs
One solution is the use of (lossless) data compression:

• it has shown great potential in other codes
• it is easy to implement
• it does not impact the physics simulated

References

• Optimizing Data Transfer Using Lossless Compression with NVIDIA nvcomp