Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Check basal melt code / runoff diagnostics #1

Closed
adele-morrison opened this issue Jul 19, 2022 · 65 comments
Closed

Check basal melt code / runoff diagnostics #1

adele-morrison opened this issue Jul 19, 2022 · 65 comments

Comments

@adele-morrison
Copy link
Collaborator

Are there any simple tests / plots we can do to show the code is working as expected?

e.g. Plot depth integrated runoff from output in Control and Perturbation to check they're the same.

Plot the zonally integrated runoff as a function of depth that it enters the model (is there a new diagnostic available that has this information?)

Other sanity checks to show that nothing unexpected has been changed (apart from runoff and heat flux associated with runoff)?

Other ideas?

@AndyHoggANU
Copy link
Collaborator

AndyHoggANU commented Jul 20, 2022

Maybe mean salinity integrated over the shelf? It will diverge, but it should take time for that divergence -- and should be a predictable.

Also, mean surface salinity on the shelf should be larger, immediately.

These are qualitative, more than quantitative [edited], checks...

@bkgf
Copy link
Collaborator

bkgf commented Jul 20, 2022

How about some theta-S diagrams as well. I'm still interested to see these.

@matthew-england-unsw
Copy link
Collaborator

Some zoomed in overturning circulation plots over the shelf, to see how the MW at depth is triggering an inverse circulation response there cf control.

@adele-morrison
Copy link
Collaborator Author

Lots of good ideas here, I suggest we split some of these off into separate issues, so it doesn't get too cluttered here. We can always refer back to those in the discussion here if needed.

@bkgf
Copy link
Collaborator

bkgf commented Jul 21, 2022

What is the final choice that was made on the melt code?

@pedrocol
Copy link
Owner

Thanks Adele!
So far there are three different runs

  • accessom2_10-GPC001: Tbasal=Tfreezing calculated with salinity equal to insitu value (1 year avail)
  • accessom2_10-GPC002: Tbasal=Tfreezing calculated with salinity equal to zero (1 year avail)
  • accessom2_10-GPC003: Tbasal calculated = min(Tfreezing,Tinsitu), with Tfreezing as in GPC001. This is done in order to avoid warming the ocean. (3 years)
    Probably the most correct one is accessom2_10-GPC003, but if we want to compare heat fluxes we should keep the original definition Trunoff = max(0,Tsurface)

@adele-morrison adele-morrison changed the title Check basal melt code Check basal melt code / runoff diagnostics Jul 28, 2022
@adele-morrison
Copy link
Collaborator Author

The total runoff in the perturbation (from the basal_fwflx diagnostic) is ~2% less than the control runoff. Why?

Need to look at the code, check what that new diagnostic is saving and what runoff is being added in.

Screen Shot 2022-07-28 at 10 29 55 am

Also there is a weird temporal variation in that ~2% value that resets on restarts and then trends downward. But at a later period is different again:

Screen Shot 2022-07-28 at 10 30 08 am

@adele-morrison
Copy link
Collaborator Author

I looked into this a little more and turns out the plot above is the sum of basal_fwflx for all layers except the top ocean layer. When you include the surface it looks even crazier like this:
Screen Shot 2022-07-28 at 1 07 52 pm

So I think our first step here should be to check in the code what the basal_fwflx diagnostic is and why it might be so large.

@adele-morrison
Copy link
Collaborator Author

Yep, this may explain the unexpected result of the overturning decreasing when we were expecting it to increase. Probably worth holding off on any other analysis until we get to the bottom of this I reckon.

@matthew-england-unsw
Copy link
Collaborator

Yes agreed. Thanks for digging into the data and uncovering.

@pedrocol
Copy link
Owner

pedrocol commented Jul 28, 2022

I deleted my previous comment, please disregard it. A few months ago, I did a few changes in the initialization of the code and then continued the run since February 2153, most likely there was a bug that I was not aware of. The strange and large values are no longer present in the output after February 1953. See below
Screenshot from 2022-07-28 18-14-54

The 2% issue is still there, but it is a constant value in time (0.02207607). I'm wondering if the first or the last level in the iteration is not being taken into account...

@pedrocol
Copy link
Owner

pedrocol commented Jul 28, 2022

I just did a print to check this and the iteration seems to work fine, I'll keep digging. Some details below
Screenshot from 2022-07-28 20-45-34

@adele-morrison
Copy link
Collaborator Author

Naive question, but where are the basal melt values coming from? Are you taking them from the control runoff or from the data in /home/552/pc5520/forcing_files/basal_fw.nc ?

@pedrocol
Copy link
Owner

The code has both options for import, for this particular case the data is set in ocean_sbc.F90, all runoff values to the south of -60 (line 3615)

@pedrocol
Copy link
Owner

I just exported the 2d basal (fwfisf variable in the MOM code, fw flux that is distributed at depth), it is in /g/data/e14/pc5520/access-om2/archive/01deg_jra55v13_ryf9091_rerun_for_easterlies/accessom2-GPC003/output1034/ocean/rregionocean_monthly_2d_basal.nc (basal_fwflx2d)
When compared with the original runoff is basal2d/runoff_con=0.99999994

@pedrocol
Copy link
Owner

pedrocol commented Jul 29, 2022

Good news, 2% problem solved. The issue was coming from the input data of max and min depth. There was a specific region where the max depth was smaller than the min depth, and therefore the iteration was not properly done. I fixed this and now the ratio between the basal melt and the runoff is 1.
Screenshot from 2022-07-29 18-52-45
I'll put below some comments on how I find out this problem and how I fixed it. Do you want me to start the run from scratch?

@adele-morrison
Copy link
Collaborator Author

Nice one Pedro! Yes, I reckon start from scratch and run another 5 years. Maybe start the other 2 perturbations from scratch too and do 2 years each of them also.

Are you happy for this to run on e14 hours @matthew-england-unsw ?

@matthew-england-unsw
Copy link
Collaborator

Yes absolutely thanks Adele.
Pedro — go for it. 👍

@aekiss
Copy link
Collaborator

aekiss commented Jul 30, 2022

Great sleuthing Pedro!

@StephenGriffies
Copy link
Collaborator

StephenGriffies commented Jul 30, 2022 via email

@pedrocol
Copy link
Owner

pedrocol commented Aug 1, 2022

Thanks a lot everyone! Is great to see all this comments.

Sounds good, I'll run a few months, and if everything is fine, I'll go for the whole 5 years.

@adele-morrison
Copy link
Collaborator Author

What do we think about only putting the basal melt (not calving) at depth if Pedro is starting from scratch again?

That way it would be more comparable to a possible later run with icebergs spread laterally at the surface. If we go this route, what’s the path to the forcing (v1.4?) that has the calving / basal split @aekiss ? Also would be good to check that the total runoff in this new version is identical to the v1.3 control.

@bkgf
Copy link
Collaborator

bkgf commented Aug 1, 2022 via email

@matthew-england-unsw
Copy link
Collaborator

matthew-england-unsw commented Aug 1, 2022

Good point Adele, I agree that's a more logical next step when the final step is basal at depth, and icebergs distributed.
But I think we have the CPU time to do all 4 cases below, which may help us understand the overall response a bit better?

  • All runoff at surface
  • All runoff at depth
  • Runoff partitioned into surface and 'at depth' amounts
  • Expt above [+iceberg distribution)

@aekiss
Copy link
Collaborator

aekiss commented Aug 1, 2022

The infrastructure is there, but we've never tried forcing MOM with separate liquid and solid runoff. Not sure what unforeseen issues we'd encounter.

Both v1.4 and v1.5 of JRA55-do have separate liquid and solid runoff; these are combined into one liquid field for earlier versions. Currently all configurations have used one of these two arrangements:

models-diagram-clip

For v1.4 and v1.5, liquid and solid runoff are provided separately to CICE, which then combines them into a single field of liquid runoff before passing to MOM. So what MOM sees is effectively the same as what was done with v1.3. There are solid runoff and heat fields passed through the coupler, but they're currently unused. More details here: COSIMA/access-om2#155

If we're thinking of going to v1.4.0, we could consider the latest 1.5.0 which fixes a few errors in v1.4.0, and has an extension (v1.5.0.1) that's continuously updated to nearly the present day: COSIMA/access-om2#247
However we haven't used v1.5.0 extensively yet. We don't have control runs to compare to so we'd need to create our own. Nor do we have v1.5.0 RYF files (though that could be done easily enough).

Data locations are:
JRA55-do v1.3.0* RYF: /g/data/ua8/JRA55-do/RYF/v1-3/
JRA55-do v1.3.0* IAF: /g/data/ua8/JRA55-do/v1-3
JRA55-do v1.3.1* IAF: /g/data/qv56/replicas/input4MIPs/CMIP6/OMIP/MRI/MRI-JRA55-do-1-3
JRA55-do v1.4.0 RYF: /g/data/ik11/inputs/JRA-55/RYF/v1-4
JRA55-do v1.4.0 IAF: /g/data/qv56/replicas/input4MIPs/CMIP6/OMIP/MRI/MRI-JRA55-do-1-4-0
JRA55-do v1.5.0 IAF: /g/data/qv56/replicas/input4MIPs/CMIP6/OMIP/MRI/MRI-JRA55-do-1-5-0
JRA55-do v1.5.0.1 IAF: /g/data/ik11/inputs/JRA-55/JRA55-do-1-5-0-1
* I think!

All current configurations of ACCESS-OM2 use v1.4.0, e.g. https://github.com/COSIMA/01deg_jra55_ryf/tree/master ; config.yaml and atmosphere/forcing.json show how the forcing files are accessed.
There are also v1.5.0 configs in the jra55v150 branch in the IAF repos, e.g. https://github.com/COSIMA/01deg_jra55_iaf/tree/jra55v150

Differences between JRA55-do versions are documented in section C here.
Depoorter et al., 2013 is used for liquid and solid (i.e. basal and calving) runoff from Antarctica in v1.3, 1.4 and 1.5. However, Greenland runoff differs between v1.3 and v1.4.

@adele-morrison
Copy link
Collaborator Author

I was thinking just to use the liquid / solid runoff from v1.4, but keep the rest of the forcing from v1.3 so the control doesn’t have to be rerun.

@aekiss
Copy link
Collaborator

aekiss commented Aug 1, 2022

As far as I can see from the documentation, runoff from Canadian and Greenland glaciers is the only difference between v1.3 and v1.4, so the spinup might not be very different.

On the other hand, v1.3 does supply the Depoorter (2013) calving flux licalvf around Antarctica in /g/data/qv56/replicas/input4MIPs/CMIP6/OMIP/MRI/MRI-JRA55-do-1-3/landIce/yrC/licalvf/gn/v20180412/licalvf_input4MIPs_atmosphericState_OMIP_MRI-JRA55-do-1-3_gn_2007-2008-clim.nc, so we could stick with v1.3 and use this file for solid runoff, so long as we also subtract it from the combined liquid+solid runoff friver. Note that in the NH the runoff friver would still be liquid+solid, since licalvf is zero there.

@adele-morrison
Copy link
Collaborator Author

adele-morrison commented Aug 1, 2022 via email

@pedrocol
Copy link
Owner

pedrocol commented Aug 1, 2022

@aekiss If I'm not wrong the calving field is not used (calving is zero everywhere) and all the fresh water flux is present in the runoff variable. If this is the case we would need to import that licalvf variable from the file you mention.
@adele157 Once the variable is in the model grid is very easy to split variables

@pedrocol
Copy link
Owner

pedrocol commented Aug 4, 2022

I went for a very pragmatic option, I took the runoff box and put the calving inside:
Screenshot from 2022-08-03 21-55-14
A few cosmetics afterwards:
ncrename -v friver,licalvf RYF.runoff_to_calving_all.1990_1991.nc
ncatted -a standard_name,licalvf,o,c,'water_flux_into_sea_water_from_calving' RYF.runoff_to_calving_all.1990_1991.nc
ncatted -a long_name,licalvf,o,c,'Water flux into sea water from calving' RYF.runoff_to_calving_all.1990_1991.nc
and now our calving field is in.

Except, I'm getting another error: assertion failed: accessom2_sync_config incompatible config between atm and ice: num_atm_to_ice_fields
Details in /home/552/pc5520/access-om2/control/01deg_jra55v13_ryf9091_rerun_for_easterlies/access-om2.err

@aekiss
Copy link
Collaborator

aekiss commented Aug 4, 2022

This JRA55-do v1.3-based config needs to be updated to what was done for the v1.4 configs so it couples licalvf from YATM to CICE to MOM.

I think you'll only need to fix these files

atmosphere/forcing.json
ice/input_ice.nml
namcouple
ocean/input.nml

You can use these changes as a guide COSIMA/01deg_jra55_ryf@v1.3.1...v1.4.0 - just make the changes that involve licalvf and see how you go.

@aekiss
Copy link
Collaborator

aekiss commented Aug 4, 2022

Also, I was hoping to use git to see the changes you're making, but I get

error: Could not read 7188e6d5aa0a6a208975b13350885fdad5e7998e

- what happens when you do git status ? does it work for you?
If not, your git repo might be broken. Did you use --no-hardlinks when you cloned /home/157/akm157/access-om2/01deg_jra55v13_ryf9091_rerun_for_easterlies? It avoids permission & corruption problems when cloning from somebody else's repo.

@pedrocol
Copy link
Owner

pedrocol commented Aug 4, 2022

This JRA55-do v1.3-based config needs to be updated to what was done for the v1.4 configs so it couples licalvf from YATM to CICE to MOM.

I think you'll only need to fix these files

atmosphere/forcing.json
ice/input_ice.nml
namcouple
ocean/input.nml

You can use these changes as a guide COSIMA/[email protected] - just make the changes that involve licalvf and see how you go.

I modified atmosphere/forcing.json, but the current version of the code does not have a fields_from_atm variable, it looks it is hard coded: cice5/build_auscom_3600x2700_722p/cpl_interface.f90
Screenshot from 2022-08-04 19-11-22

@pedrocol
Copy link
Owner

pedrocol commented Aug 4, 2022

Screenshot from 2022-08-04 19-14-25

@aekiss
Copy link
Collaborator

aekiss commented Aug 5, 2022

Looks like you're using code from late 2019
https://github.com/COSIMA/access-om2/tree/7f834a0
So it's missing the updates needed for JRA55do v1.4.

Is this what you've based your MOM edits on?

I'm in the middle of updating libaccessom2, cice and the standard configs and executables, so there should be newer versions available sometime next week which would be a better place to start from.

@bkgf
Copy link
Collaborator

bkgf commented Aug 5, 2022 via email

@pedrocol
Copy link
Owner

pedrocol commented Aug 5, 2022

Yes, I'm using the same code that Adele used for her run /home/157/akm157/access-om2/source_code/ (for comparison ends, and because I was not able to use the restarts in the newest version of the code).

The Basal melt parameterization is coded in the latest version of the code, and I adapted this for the late 2019 version of the code.

@pedrocol
Copy link
Owner

pedrocol commented Aug 5, 2022

@bkgf The basal melt is added in the same way as the runoff is, as temperature and salinity fluxes, and in a separate routine as a mass/volume flux (exactly as runoff)

@aekiss
Copy link
Collaborator

aekiss commented Aug 5, 2022

Hm, ok. This idea of splitting basal and calving fluxes is starting to feel like a rabbit-hole... It looks like you'd need new branches of the code (cice, and possibly libaccessom2 and mom) based on your old versions but with cherry-picked bits from newer code, which seems error-prone and kludgey. It would be cleaner and easier to do the split based on the established v1.4 (or 1.5) configs and code, where these problems have already been resolved. It could then be adapted to use this hybrid v1.3 + licalvf if desired.

What error do you get when you try running the new code with your old restarts?

@pedrocol
Copy link
Owner

pedrocol commented Aug 5, 2022

I agree, it looks very complicated. Probably the easiest will be to make an offline interpolation of the calving field in the MOM grid, and import this in ocean_sbc.F90. But this is only a workaround, it won't be useful for newer runs.

This was several months ago (February), I lost track of the errors, I only got this (see below) from a mail. If I remember well and I'm not wrong, I think the newer version of the code uses more tracers (co2? maybe) than the previous version, and the restarts don't have these fields.

forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
fms_ACCESS-OM-BGC 0000000001CACBA4 Unknown Unknown Unknown
libpthread-2.28.s 000014592E5AFC20 Unknown Unknown Unknown
libuct_ib.so.0.0. 0000145913030697 Unknown Unknown Unknown
libucp.so.0.0.0 0000145914C20E9A ucp_worker_progre Unknown Unknown
mca_pml_ucx.so 00001459150C8397 mca_pml_ucx_progr Unknown Unknown
libopen-pal.so.40 00001459298BB72B opal_progress Unknown Unknown
libmpi.so.40.20.2 000014592EB8E2D5 ompi_request_defa Unknown Unknown
libmpi.so.40.20.2 000014592EBD113F PMPI_Waitall Unknown Unknown
libmpi_mpifh.so 000014592EEC67ED Unknown Unknown Unknown
fms_ACCESS-OM-BGC 0000000001B48A00 Unknown Unknown Unknown
fms_ACCESS-OM-BGC 0000000001B479B9 Unknown Unknown Unknown
fms_ACCESS-OM-BGC 0000000001A55993 mod_oasis_advance 1130 mod_oasis_advance.F90
fms_ACCESS-OM-BGC 0000000001957178 mod_oasis_getput_ 760 mod_oasis_getput_interface.F90
fms_ACCESS-OM-BGC 0000000000411BCC mom_oasis3_interf 688 mom_oasis3_interface.F90
fms_ACCESS-OM-BGC 00000000004239A8 MAIN__ 637 ocean_solo.F90
fms_ACCESS-OM-BGC 000000000040F6A2 Unknown Unknown Unknown
libc-2.28.so 000014592E1FB493 __libc_start_main Unknown Unknown
fms_ACCESS-OM-BGC 000000000040F5AE Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)

@adele-morrison
Copy link
Collaborator Author

That does sound like it's getting complicated. Given that there are so many assumptions already with this method of putting the basal melt at depth, how about we just go with something simple, like putting 50% of the runoff at depth and leaving the other 50% at the surface. The 50% is an approximation of the net circumpolar fluxes from Depoorter et al. 2013 (52% basal melt, 48% calving, but these have ~4% uncertainties).

@aekiss
Copy link
Collaborator

aekiss commented Aug 5, 2022

There are now 2 types of ACCESS-OM2 MOM executables, one including BGC (fms_ACCESS-OM-BGC_*_libaccessom2_*.x, which is what you tried to use), and one without (fms_ACCESS-OM_*_libaccessom2_*.x). If I recall correctly, a non-BGC executable won't need extra fields in the restarts. You can build the non-BGC version by setting export mom_type=ACCESS-OM in install.sh.

@pedrocol
Copy link
Owner

pedrocol commented Aug 5, 2022

@aekiss I will open this in a new issue, since we have to make the import of the calving field work and we also have to make the restarts work in the new version of the code. These two aspects may be out of the scope of the validation of the basal melt parameterization

@matthew-england-unsw
Copy link
Collaborator

Agreed @adele157 that does sound like a sensible starting point.

@aekiss
Copy link
Collaborator

aekiss commented Aug 5, 2022

Yes @adele157 that sounds a lot easier - it amounts to having a nonuniform injection of runoff with depth. Perhaps the split fraction could be a namelist parameter?

@aekiss
Copy link
Collaborator

aekiss commented Aug 5, 2022

Thanks @pedrocol - maybe 2 new issues would be even better, as calving import and restarts are independent problems

@pedrocol
Copy link
Owner

pedrocol commented Aug 5, 2022

Great, I'll put this to run, we may have a couple of years for the next meetings.
@aekiss A namelist parameter for the ratio sounds like a good idea

@aekiss
Copy link
Collaborator

aekiss commented Aug 5, 2022

A constant split is a good place to start, but pretty rough - Depoorter et al fig 1 show the ratio depends strongly on location.

@aekiss
Copy link
Collaborator

aekiss commented Aug 10, 2022

Probably a bit late to say this, but something other than a 50/50 split (say, 55/45) would be better for testing, as it will show you if you've accidentally swapped variables anywhere.

@pedrocol
Copy link
Owner

pedrocol commented Aug 11, 2022

Screenshot from 2022-08-10 22-04-34

Here is the piece of code that performs the distribution at depth and applies this to the tendency. If the amount of runoff present in the basal3d variable is ok, as we have seen in the validation plots, normally the code should behave correctly.

As an additional check I put below a mass budget of the basal melt param run:
Screenshot from 2022-08-10 22-09-18

and for the control run

Screenshot from 2022-08-10 22-17-19

If we add the calving, basal and river, of the basal melt param run, and compare this with the control run, the numbers are almost the same (precision question aside).
I put the mass budget only since the salt budget is zero, and the heat budget is different since we use a different definition for the basal temperature.

@StephenGriffies
Copy link
Collaborator

StephenGriffies commented Aug 18, 2022

These are encouraging...single time step diagnostics appear fine for mass. Here are further diagnostics that we should document:

--single time step diagnostics for heat. We should be sure the heat content impacts from basal melt are properly accounted for.

--single time step diagnostics for salt. Although there is no salt in the basal melt, I still wish to see the salt diagnostics to be sure everything is fine.

--multiple time step diagnostics for heat and mass and salt. The multiple time step diagnostics sometimes pick up mistakes that are missed by the single time step diagnostics. The model has a default for the number of time steps it integrates over; I think it is 100. Just need to be sure the number of time steps is not more than the length of the run.

@pedrocol
Copy link
Owner

Thanks for your comment Steve. In order to be able to compare those variables I have to run the code in the same way as the control run, this is, distributing the basal in the first 40 meters and using the same temperature for the runoff (tbasal=trunoff), right?

@StephenGriffies
Copy link
Collaborator

I am not looking for exactly reproducing an earlier run. Just wish to look at order of magnitudes for the diagnostics that check for conservation. If the orders are roughly the same, then we are fine. So any run should be sufficient.

@adele-morrison
Copy link
Collaborator Author

adele-morrison commented Oct 11, 2022 via email

@pedrocol
Copy link
Owner

pedrocol commented Oct 14, 2022 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants