Exact restart failure for f45_g37 grid for SOIL2N_TNDNCY_VERT_TRANS for comprehensive FATES testmod

[glemieux]

I'm noticing an interesting RMS diff with the fates_cold_allvars ERP testmod results. It fails COMPARE_base_rest, but only on a single output:

 SOIL2N_TNDNCY_VERT_TRANS   (lon,lat,levdcmp,time)  t_index =     10    10
          1    49680  (    30,    19,     2,     1) (    30,    19,     1,     1) (    11,    24,     1,     1) (    11,    24,     1,     1)
               21990   9.866014255566213E-12         -1.555327221181013E-11 1.7E-24 -1.576961273444979E-17 4.8E-12 -1.576961273444979E-17
               21990   9.866014255566213E-12         -1.555327221181013E-11         -1.576961438881101E-17         -1.576961438881101E-17
               49680  (    30,    19,     2,     1) (    30,    19,     1,     1)
          avg abs field values:    1.132267023196132E-13    rms diff: 0.0E+00   avg rel diff(npos):  4.8E-12
                                   1.132267023196132E-13                        avg decimal digits(ndif):  7.0 worst:  7.0
 RMS SOIL2N_TNDNCY_VERT_TRANS         0.0000E+00            NORMALIZED  0.0000E+00

Looking through the e3sm_land_developer test output, it appears that the only other testmod that outputs this variable is the SMS smallville test.

Originally posted by @glemieux in #6018 (comment)

[glemieux]

Excluding this output variable results in the test passing for this particular resolution even on longer timescales. As no other ERS diffs crop up, my assumption is that this issue is isolated to this particular history output.

The issue also appears to geographically isolated, only showing up in one area:

Based on that insight, I tested the same configuration, but with a different resolution, this time using the ne4pg2_ne4pg2 grid. The issue did not present itself with this resolution.

[rgknox]

needle in one of a thousand haystacks

[glemieux]

needle in one of a thousand haystacks

Truth. I should also note that it was only showing up for one decomp dimension as well.

[glemieux]

Interestingly (weirdly), a longer 30 day test does not result in the exact restart failure; everything passes. I'm thinking we should just call this a glitch of some sort and close this out for now. We can open it again if it crops up with a different length test.

[glemieux]

Reopening due to reappearance with #6235

[glemieux]

I wanted to highlight here this comment from #6235 that may be relevant for future work to address this:
#6235 (comment)