ML Preprocessing Module

[jejjohnson]

This will define some factors that will be important for the ML method where we define the input, output and meta. For example, we may want to input a batch of coordinates or patches from a cube. This can potentially be a difficult step due to the “heaviness” of the data.

---

Case Studies:

• Interpolation: Batch of coordinates
• Interpolation: Batch of patches, x=ssh_sims(lat x lon x time), y=ssh_obs(lat x lon x time),
• Surrogate: Batch of patches, x=ssh(lat x lon x t-1), y=ssh(lat x lon x t)

1. Patch-ify, e.g. [h,w] -> [P, h//factor, w//factor]
2. Cube-ify, e.g. [h,w,c] -> [P, c//factor, h//factor, w//factor]
3. Coordinate-ify, e.g. xr.dataset -> pd.DataFrame, [LAT,LON,TIME,SSH] -> [ssh,lat,lon,time]*(latxlonxtimex1)
   1. broadcast_like
   2. ravel
   3. storage suited for local sampling!, e.g. 1) storage for frequency sampling, 2) storage for local sampling
4. Save sub-patches/cubes, e.g. zarr (Examples - ex1a| ex1b | ex2a | ex2b
5. Save coordinates, e.g. csv, parquet (Examples - Kaggle | Nvidia-merlin | nvtabular