You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I should say up-front that I'm not sure if this is actually a bug with aoflux_grid=xgrid, but it is a difference between aoflux_grid=xgrid and aoflux_grid=ogrid, and it seems like it might be contributing to a model crash. So I'm opening this to start a discussion on whether this might be a bug.
I'll start with the conclusion before diving into details: It seems like, with aoflux_grid=ogrid, we get 0 values sent to atm from the atm-ocn flux calculations over land points, but NOT over 100% ice points; but with aoflux_grid=xgrid, we get 0 values sent to atm from the atm-ocn flux calculations over any points with (lnd+ice)=1 (i.e., any points with ofrac=0). This MAY be contributing to a crash in some configurations with xgrid, though I'm not sure yet if that's the cause. @jedwards4b@mvertens@uturuncoglu@DeniseWorthen - do any of you have a sense of what's right here?
This started with an investigation of a failure in the test SMS_D_Ln9.f09_f09_mg17.FCnudged_GC.derecho_intel.cam-outfrq9s. I'm running this from cesm3_0_alpha03c but with CMEPS updated to cmeps1.0.18 (which is needed to fix some other issues with the exchange grid). Out-of-the-box, this test passes. However, when adding the following to user_nl_cpl – aoflux_grid = "xgrid" – this test fails with a divide by zero in drydep_mod:
This test passes with aoflux_grid = "ogrid", and a similar test without chemistry – SMS_D_Ln9.f09_f09_mg17.FHIST.derecho_intel.cam-outfrq9s – passes even with xgrid.
I haven't dug deeply into the relevant CAM code, but I decided to look for differences in variables sent to the atmosphere in the test SMS_D_Ln9.f09_f09_mg17.FHIST.derecho_intel.cam-outfrq9s with xgrid vs. ogrid. I started by looking at ustar, since that's one of the terms in the line with divide-by-zero (though note that I have not confirmed that this is the term causing the problem).
Here is ustar sent to atm using xgrid:
And here using ogrid:
Both have 0 values over land, but note that the xgrid run has additional 0 values in the Arctic ocean and near Antarctica. I see these same extra 0 values in Med_aoflux_atm_So_ustar, but not in Med_aoflux_ocn_So_ustar. I spot-checked some other fields from the atm-ocn flux calculation and see the same thing with other fields.
Here is a map showing where ofrac is essentially 0:
And here are points where ifrac is essentially 1:
By eye, these seem to match up very well with the grid cells that have 0 values in the run with aoflux_grid=xgrid. This leads me to the conclusion at the top of the issue.
The text was updated successfully, but these errors were encountered:
After some consultation with @mvertens and additional testing, we feel that xgrid is working as intended here. The 0 values over sea ice grid cells also appear in runs with aoflux_grid = "xgrid" but where the atmosphere and ocean are running on different grids. CAM crashes in the same way in SMS_D_Ln9.f09_g17.FCnudged_GC.derecho_intel.cam-outfrq9s with aoflux_grid = "ogrid" as I noted above with SMS_D_Ln9.f09_f09_mg17.FCnudged_GC.derecho_intel.cam-outfrq9s with "xgrid". I believe this is a CAM issue, unless CAM wants to push for changes in the long-standing behavior of the mediator in this respect (which could be a reasonable solution). So I have moved this to a CAM issue:
I should say up-front that I'm not sure if this is actually a bug with aoflux_grid=xgrid, but it is a difference between aoflux_grid=xgrid and aoflux_grid=ogrid, and it seems like it might be contributing to a model crash. So I'm opening this to start a discussion on whether this might be a bug.
I'll start with the conclusion before diving into details: It seems like, with aoflux_grid=ogrid, we get 0 values sent to atm from the atm-ocn flux calculations over land points, but NOT over 100% ice points; but with aoflux_grid=xgrid, we get 0 values sent to atm from the atm-ocn flux calculations over any points with (lnd+ice)=1 (i.e., any points with ofrac=0). This MAY be contributing to a crash in some configurations with xgrid, though I'm not sure yet if that's the cause. @jedwards4b @mvertens @uturuncoglu @DeniseWorthen - do any of you have a sense of what's right here?
This started with an investigation of a failure in the test SMS_D_Ln9.f09_f09_mg17.FCnudged_GC.derecho_intel.cam-outfrq9s. I'm running this from cesm3_0_alpha03c but with CMEPS updated to cmeps1.0.18 (which is needed to fix some other issues with the exchange grid). Out-of-the-box, this test passes. However, when adding the following to user_nl_cpl –
aoflux_grid = "xgrid"
– this test fails with a divide by zero in drydep_mod:which is here:
As a side-note: This test also fails in non-debug mode, though more cryptically (so I'm not positive what's going on here):
This test passes with
aoflux_grid = "ogrid"
, and a similar test without chemistry –SMS_D_Ln9.f09_f09_mg17.FHIST.derecho_intel.cam-outfrq9s
– passes even with xgrid.I haven't dug deeply into the relevant CAM code, but I decided to look for differences in variables sent to the atmosphere in the test
SMS_D_Ln9.f09_f09_mg17.FHIST.derecho_intel.cam-outfrq9s
with xgrid vs. ogrid. I started by looking at ustar, since that's one of the terms in the line with divide-by-zero (though note that I have not confirmed that this is the term causing the problem).Here is ustar sent to atm using xgrid:
And here using ogrid:
Both have 0 values over land, but note that the xgrid run has additional 0 values in the Arctic ocean and near Antarctica. I see these same extra 0 values in Med_aoflux_atm_So_ustar, but not in Med_aoflux_ocn_So_ustar. I spot-checked some other fields from the atm-ocn flux calculation and see the same thing with other fields.
Here is a map showing where ofrac is essentially 0:
And here are points where ifrac is essentially 1:
By eye, these seem to match up very well with the grid cells that have 0 values in the run with aoflux_grid=xgrid. This leads me to the conclusion at the top of the issue.
The text was updated successfully, but these errors were encountered: