Memory Usage: Add memory transferred between Kokkos Mem Spaces

[vlkale]

This PR adds memory transferred between Kokkos Mem Spaces to the Kokkos Tools memory-usage tool library. This is done based on a request in Kokkos Tools Github Issue #50.

This PR adds functionality to the tool so that the size of data transferred in the deep_copy in a Kokkos application program is accumulated during the execution of a Kokkos program. The deep_copy accumulation is done per Kokkos Memory Space dst->src pair (e.g., OpenMP on host CUDA on device). Note that the "dst" and "src" Kokkos memory space being the same means that there was a deep_copy in the same memory space.

When Kokkos is finalized in a Kokkos application program, the finalize callback prints out the deep_copy accumulations per memory space to the file.

[vlkale]

Here is output from the stream.cuda in the Kokkos-core benchmarks directory run on Perlmutter on 1 GPU. The number of bytes in the HighWater at the last point in time shown in the first table is equal to the data transferred between host and device.

-----@nid----:~/kks-clean/benchmarks/stream> cat nid001213-1948341-Cuda.memspace_usage
# Space Cuda
# Time(s)  Size(MB)   HighWater(MB)   HighWater-Process(MB)
0.018076 7629.4 7629.4 142.7
0.047229 15258.8 15258.8 143.5
0.071899 22888.2 22888.2 144.0
12.252089 15258.8 22888.2 23032.6
12.313410 7629.4 22888.2 23032.6
12.331277 0.0 22888.2 23032.6
# Data transferred between Kokkos Memory Spaces --- 
# DstSpace     SrcSpace    Data-Transferred(MB)
Cuda Cuda 0.0 
# DstSpace     SrcSpace    Data-Transferred(MB)
Cuda Host 22888.2 
# DstSpace     SrcSpace    Data-Transferred(MB)
Host Cuda 22888.2 
# DstSpace     SrcSpace    Data-Transferred(MB)
Host Host 0.0 

------@nid----:~/kks-clean/benchmarks/stream> cat nid001213-1948341-
nid001213-1948341-Cuda.memspace_usage  nid001213-1948341-Host.memspace_usage  
vkale3@nid001213:~/kks-clean/benchmarks/stream> cat nid001213-1948341-Host.memspace_usage 
# Space Host
# Time(s)  Size(MB)   HighWater(MB)   HighWater-Process(MB)
0.077138 7629.4 7629.4 144.0
0.768111 15258.8 15258.8 7773.6
1.455819 22888.2 22888.2 15403.0
12.218183 15258.8 22888.2 23032.6
12.229001 7629.4 22888.2 23032.6
12.240329 0.0 22888.2 23032.6
# Data transferred between Kokkos Memory Spaces --- 
# DstSpace     SrcSpace    Data-Transferred(MB)
Cuda Cuda 0.0 
# DstSpace     SrcSpace    Data-Transferred(MB)
Cuda Host 22888.2 
# DstSpace     SrcSpace    Data-Transferred(MB)
Host Cuda 22888.2 
# DstSpace     SrcSpace    Data-Transferred(MB)
Host Host 0.0