Model comparison

[galengorski]

This discussion is about comparing model runs and visualizing results. All models are run using the NHD fabric on the medium observed data which sets a threshold of at least 100 DO observations per site, and results in 14 sites.

Sites

• Training sites: '01466500', '01472119', '01473499', '01473675', '01473780', '01474500', '01480617', '01480870', '01481000', '01481500'
• Validation sites : "01472104", "01473500", "01481500"
• Test sites: "01475530", "01475548"

Models

• (0) baseline: predict do_min, do_mean, and do_max
• (1a) multitask: predict do_min, do_mean, do_max, GPP, and ER (also predict K, Temp, Depth but don't include in loss func)
• (1b) metab_multitask: predict do_min, do_mean, do_max, GPP, ER, K, Temp, Depth
• (2) metab_dense: predict GPP, ER, K, Temp, Depth use dense layer to then predict do_min, do_mean, do_max,

[galengorski]

I think that a good place to start is to compare the baseline model (0) to the metab_dense model (2), as the metab dense model explicitly routes the predictions through the metabolism outputs it seems like the model that relies on the physical processes most. Here is a plot of the RMSE by site comparing models 0 and 2:

[jsadler2]

Very interesting, @galengorski.

Did you run a number or replicates or just one for this comparison?

[galengorski]

This is a single replicate for each model, I am trying to log onto tallgrass to run more replicates, but I can't get a node. I think running more replicates would help us determine:

1. how robust these patterns are
2. if the wintertime drop off is consistent between replicates for 01481500 as seen below
3. how adding in tmmn as a predictor affected the output

[galengorski]

The site that improves the most using the metab_dense model is 01481500 which is on Brandywine Creek at Wilmington, DE. Here is a time series of that site:

It looks like the baseline model is predicting a big anomaly in winter that doesn't exist, and the metab_dense model is not, that probably accounts for some of the model improvement. Other than that anomaly it actually looks like the baseline model does better at this site particularly with do_max. Also meta_dense under predicts do_min worse than the baseline, although both under predict the troughs.

[jsadler2]

That is super weird that each winter the 0 model is predicting that big drop-off. It seems pretty suspicious to me, like there's some weird discontinuity in the input data somehow ... but just at this site, because it doesn't show up in any of the other sites. Also, it looks like it happens at the exact same time each year, which also makes me suspicious.

But the 0 model and the 2 model should be using the same input data right? When I look back at the code, I don't see how they could be different, but it may be worth just checking the "{outdir}/prepped.npz" files for both the 0 and the 2 to make sure the inputs are the same.

[galengorski]

Yeah good idea. That's also why I plotted the dynamic inputs below to see if there was an anomaly in the input data, but there isn't anything that I can see.

[galengorski]

This is the same style time series from site 01466500, the site on the coastal plain that is draining wetlands essentially:

Interestingly, it looks like the metab_dense model does a better job with the lowest values, such as the troughs in do_min. The baseline model looks like it pegs out, but the metab_dense model seems to follow the curve much better. It also looks like it tracks the rising limb out of the trough much better.

[galengorski]

Just for fun here is another one, this is for site 01472104 which is on the Schuylkill River above Philadelphia:

This one also shows an improvement in the bias, particularly for do_mean, and better tracking of the rising limb, particularly in do_min.

[galengorski]

Adding in tmin as a dynamic predictor

The plots above were made using tmax, precip, and solar radiation as the dynamic predictors. Adding in tmin doesn't have a huge effect, but it does seem to improve the validation sites for do_min and do_max in general:

Might be easier to see the differences when the sites are aggregated together:

[galengorski]

My thought was that adding tmin would have the greatest effect on do_min because tmin would set the baseline for how much ER occurred during the night time? Not sure if that is the right reasoning, but then I looked at how the addition of tmin affected the prediction of metab model outputs. This could only be done for the metab_dense model:

So adding tmin improves prediction of ER, but decreases performance for GPP across the board, and is pretty mixed for the other variables.

[galengorski]

Also, in this run the large anomaly in the baseline model for site 01481500 was not present. This makes me want to run more replicates and compare ensembles instead of single runs:

[lekoenig]

Following up on this, I ran 5 replicates of the baseline LSTM model with and without tmmn as a predictor. Here's the RMSE ("val" partition only) from the overall_metrics file, so averaging across observations and sites:

The global RMSE values suggest that adding tmmn probably results in a small improvement across the three target variables, perhaps more so for DO-min and DO-max (consistent w/ Galen's assessment from 1 rep). In our meeting on 10/5/22, I mentioned that it seemed like the model w/ tmmn was actually performing worse for 4 out of the 6 validation sites based on 1 replicate model run. After running more reps, I don't think the tmmn model necessarily performs worse but rather, the differences are generally small relative to the variability among model reps. What stands out to me is the large variability among reps for 01481500 for both versions of the model.

Here's the timeseries for 01481500 (a val site on Brandywine Creek) where each line shows the predictions for a different model rep. In one of the baseline reps we see the excursions that Galen highlighted above, so that's contributing to the error for the baseline model. Both models are really struggling to get the magnitude of the summer minima and for some reps, the tmmn model is more biased than the baseline model during those times.

I was hoping to get a sense for how the model might be using tmmn (aside from what is already gleaned from tmmx), so I plotted global permutation feature importance across reps as well as the expected gradients for tmmn and tmmx for one 12-month sequence (6 months are shown below; from model rep 4).

If I'm interpreting the EG plots correctly, it seems like for this site (01481500) the expected gradients for tmmx and tmmn largely overlap and that there's not a lot of unique information between them. I wonder why, then, adding tmmn affects the model predictions at this site. Alternatively, maybe the tmmn predictions are just within the [wide] range of variability we'd get for the baseline model without tmmn if we ran more reps.