Similar to Moving MNIST, we also provide the advanced version of MNIST, i.e., MFMNIST benchmark results, using $10\rightarrow 10$ frames prediction setting following PredRNN. Metrics (MSE, MAE, SSIM, pSNR) of the best models are reported in three trials. Parameters (M), FLOPs (G), and V100 inference FPS (s) are also reported for all methods. All methods are trained by Adam optimizer with Onecycle scheduler and single GPU.
- For a fair comparison of different methods, we provide config files in configs/mfmnist.
- We also benchmark popular Metaformer architectures on SimVP with training times of 200 epochs. We provide config files in configs/mfmnist/simvp.
STL Benchmarks on MFMNIST
| Method |
Setting |
Params |
FLOPs |
FPS |
MSE |
MAE |
SSIM |
PSNR |
Download |
| ConvLSTM-S |
200 epoch |
15.0M |
56.8G |
113 |
28.87 |
113.20 |
0.8793 |
22.07 |
model | log |
| ConvLSTM-L |
200 epoch |
33.8M |
127.0G |
50 |
25.51 |
104.85 |
0.8928 |
22.67 |
model | log |
| PredNet |
200 epoch |
12.5M |
8.6G |
659 |
185.94 |
318.30 |
0.6713 |
14.83 |
model | log |
| PhyDNet |
200 epoch |
3.1M |
15.3G |
182 |
34.75 |
125.66 |
0.8567 |
22.03 |
model | log |
| PredRNN |
200 epoch |
23.8M |
116.0G |
54 |
22.01 |
91.74 |
0.9091 |
23.42 |
model | log |
| PredRNN++ |
200 epoch |
38.6M |
171.7G |
38 |
21.71 |
91.97 |
0.9097 |
23.45 |
model | log |
| MIM |
200 epoch |
38.0M |
179.2G |
37 |
23.09 |
96.37 |
0.9043 |
23.13 |
model | log |
| MAU |
200 epoch |
4.5M |
17.8G |
201 |
26.56 |
104.39 |
0.8916 |
22.51 |
model | log |
| E3D-LSTM |
200 epoch |
51.0M |
298.9G |
18 |
35.35 |
110.09 |
0.8722 |
21.27 |
model | log |
| PredRNN.V2 |
200 epoch |
23.9M |
116.6G |
52 |
24.13 |
97.46 |
0.9004 |
22.96 |
model | log |
| DMVFN |
200 epoch |
3.5M |
0.2G |
1145 |
118.32 |
220.02 |
0.7572 |
16.76 |
model | log |
| SimVP+IncepU |
200 epoch |
58.0M |
19.4G |
209 |
30.77 |
113.94 |
0.8740 |
21.81 |
model | log |
| SimVP+gSTA-S |
200 epoch |
46.8M |
16.5G |
282 |
25.86 |
101.22 |
0.8933 |
22.61 |
model | log |
| TAU |
200 epoch |
44.7M |
16.0G |
283 |
24.24 |
96.72 |
0.8995 |
22.87 |
model | log |
Benchmark of MetaFormers Based on SimVP (MetaVP)
| MetaFormer |
Setting |
Params |
FLOPs |
FPS |
MSE |
MAE |
SSIM |
PSNR |
Download |
| IncepU (SimVPv1) |
200 epoch |
58.0M |
19.4G |
209 |
30.77 |
113.94 |
0.8740 |
21.81 |
model | log |
| gSTA (SimVPv2) |
200 epoch |
46.8M |
16.5G |
282 |
25.86 |
101.22 |
0.8933 |
22.61 |
model | log |
| ViT |
200 epoch |
46.1M |
16.9.G |
290 |
31.05 |
115.59 |
0.8712 |
21.83 |
model | log |
| Swin Transformer |
200 epoch |
46.1M |
16.4G |
294 |
28.66 |
108.93 |
0.8815 |
22.08 |
model | log |
| Uniformer |
200 epoch |
44.8M |
16.5G |
296 |
29.56 |
111.72 |
0.8779 |
21.97 |
model | log |
| MLP-Mixer |
200 epoch |
38.2M |
14.7G |
334 |
28.83 |
109.51 |
0.8803 |
22.01 |
model | log |
| ConvMixer |
200 epoch |
3.9M |
5.5G |
658 |
31.21 |
115.74 |
0.8709 |
21.71 |
model | log |
| Poolformer |
200 epoch |
37.1M |
14.1G |
341 |
30.02 |
113.07 |
0.8750 |
21.95 |
model | log |
| ConvNeXt |
200 epoch |
37.3M |
14.1G |
344 |
26.41 |
102.56 |
0.8908 |
22.49 |
model | log |
| VAN |
200 epoch |
44.5M |
16.0G |
288 |
31.39 |
116.28 |
0.8703 |
22.82 |
model | log |
| HorNet |
200 epoch |
45.7M |
16.3G |
287 |
29.19 |
110.17 |
0.8796 |
22.03 |
model | log |
| MogaNet |
200 epoch |
46.8M |
16.5G |
255 |
25.14 |
99.69 |
0.8960 |
22.73 |
model | log |
| TAU |
200 epoch |
44.7M |
16.0G |
283 |
24.24 |
96.72 |
0.8995 |
22.87 |
model | log |
