Create new land only initial conditions (ongoing)

[wwieder]

As we move to CTSM5.2 datasets and introduce CLM6_0 physics options, it seems like it would also be helpful to provide new initial conditions files. @dlawrenncar noted, this may be most important for coupled model simulations, so we can provide more realistic land states in F and B cases as we move forward with CESM3 development runs.

Given very high productivity biases in CRU-JRA forced runs, I'd suggest we provide 1850 initial conditions forced with GSWP3 using CTSM5.2 surface data. It may be most straight forward to do this after #2348 comes to main?

Looking through the LMWG-Dev issues, I don't think we have a GSWP3 forced run that's similar to this case, #54, do we @olyson? If not, we can create a new issue on LWMG-Dev for this.

We should make sure that isotopes, tillage, and residue removal are all on for these spin ups.

Definition of done:

• Decide on which LND_TUNING_MODES will have initial conditions (clm5_0_GSWP3, clm6_0_CRUJRA, clm5_0_cam6, clm6_0_cam7?)
• Which resolutions will be run? ne30pg3, f09, f19, f45? What about NEON sites?
• Run all the configurations needed
• Preliminary finidat file in place for clm5_1/clm6_0 all forcing options: lnd/clm2/initdata_esmf/ctsm5.2/clmi.I1850Clm60BgcCrop-ciso.1361-01-01.ne30pg3_mg17_c240317.nc
• Put files in place and in XML and into a tag

[ekluzek]

Since, this requires CTSM5.2 it actually should wait for #2372. If we need it sooner it could be added to CESM3_dev. But, I'd like to avoid that if we can. This was also on the CTSM5.2 board as a post-5.2 activity, so I'm linking this there.

Also this relates to #554, so linking it here.

[olyson]

@wwieder and I discussed this and thought that we could just clone the most recent deadveg branch simulation and switch from CRUJRA to GSWP3. In that recent simulation, tillage and residue removal was on. And I have been using CTSM5.2 surface datasets, i.e., for that run I used

/glade/campaign/cesm/cesmdata/cseg/inputdata/lnd/clm2/surfdata_esmf/ctsm5.2.0/surfdata_0.9x1.25_hist_78pfts_CMIP6_1850_c230517.nc

I know that @slevis-lmwg has recently re-generated all of the datasets, so I was planning to use that version which appears to be:

/glade/campaign/cesm/cesmdata/cseg/inputdata/lnd/clm2/surfdata_esmf/ctsm5.2.0/surfdata_0.9x1.25_hist_1850_78pfts_c240216.nc

Is there another reason to wait for #2372 ?

[ekluzek]

Nope that makes perfect sense @olyson. Better to not have to wait when you don't have to.

[olyson]

See NCAR/LMWG_dev#57

[olyson]

I initially set this up as an f09 (1deg) simulation, but I think it should actually be an ne30 since we'll use it as initial conditions for the ne30 F- and B-cases coming up. I don't think the finidat interpolation will handle going from 1deg to ne30. Is that a correct assumption @ekluzek and @billsacks ?

[ekluzek]

@olyson it should be fine to interpolate from f09 to ne30np4.pg3 grid. The interpolation is just nearest neighbor so it will work between any type of grid.

But, I agree that we should spinup the land with the workhorse ne30 grid rather than f09. We'll interpolate to f09 from it for the land only simulations.

[billsacks]

I agree with @ekluzek .

[olyson]

Here is an ne30 spunup initial file for 1850 with configuration as noted above:

/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm51_ctsm51d166deadveg_ne30pg3ne30pg3mg17_GSWP3V1_ABsnoCDE_blk_A5BCD_1850pAD.clm2.r.1361-01-01-00000.nc

[wwieder]

Thanks @olyson should this be the default finidat file we point to for simulations with 'modern' tags? Here I'm especially thinking about the CESM alpha (or beta) 17 tag that's upcoming?

[olyson]

Yes, I think it will be suitable for the upcoming CESM tags.

[wwieder]

@ekluzek and @slevis-lmwg, can we make it so this initial conditions file is used out of the box with the next round of coupled model experiments?

/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm51_ctsm51d166deadveg_ne30pg3ne30pg3mg17_GSWP3V1_ABsnoCDE_blk_A5BCD_1850pAD.clm2.r.1361-01-01-00000.nc

[wwieder]

looking at the project board, I wonder if this should come in with #2492, or actually #2501?

[ekluzek]

@wwieder it looks like that's GSWP3 forcing finidat file. Do you want that with I1850Clm60BgcCrop and also with B1850/F1850 or just the former because it's for GSWP3 forcing?

[wwieder]

all of the above B, I, & F1850 until we get new cpl.hist files to try a spinup for the B case. We want to provide living arctic vegetation as much as possible.

[ekluzek]

It looks like the fsurdat file used for that IC file was with a preliminary version of ctsm5.2.0 datasets, and is different by at least roundoff level. That means that it will likely require use_init_interp set to TRUE.

Also @olyson note that in CSEG Meeting Notes Mike Levy tells us that they will be going to the t232 mask for MOM. So we should start using it for our simulations as well.

[olyson]

Ok, sounds like it is a showstopper, so I'll start another spinup with the latest code and the latest surface dataset (which out of the box seems to be /glade/campaign/cesm/cesmdata/cseg/inputdata/lnd/clm2/surfdata_esmf/ctsm5.2.0/surfdata_ne30np4.pg3_hist_1850_78pfts_c231026.nc) and the t232 mask.

[ekluzek]

@olyson I don't mean to say it's a showstopper. I mean to say we need to transition over to it. I think we need this in place by end of June for the science capability freeze. But, actually we should talk as a group about this in CTSM as well as CESM.

[olyson]

Ah, ok, sorry, I didn't read your comments carefully enough.

[wwieder]

FWIW it also seems like @slevis-lmwg is creating some new initial condition files for matrix-CN work, although these also are unlikely to have the correct ocean mask.

[ekluzek]

New finidat file name will be:

lnd/clm2/initdata_esmf/ctsm5.2/clmi.I1850Clm60BgcCrop-ciso.1361-01-01.ne30pg3_mg17_c240317.nc

And used for standard 1850 control and hist compsets starting at 1850 with clm5_1 and clm6_0 physics.

[wwieder]

fwiw @adrifoster suggested just using the mesh_maker tool in tools/site_and_regional to generate the mesh, Sam.

[slevis-lmwg]

Thanks for the reminder @wwieder. I tried mesh_maker just now and got the same error that I saw with all my attempts. I'm starting to wonder whether the problem is with the datm data somehow.

No idea whether this would be a problem, but here's the difference between a CRUJRA file that works:

        float lon(lon) ;
                lon:units = "degrees_east" ;
                lon:long_name = "longitude" ;
                lon:mode = "time-invariant" ;
        float lat(lat) ;
                lat:units = "degrees_north" ;
                lat:long_name = "latitude" ;
                lat:mode = "time-invariant" ;
        float LONGXY(lat, lon) ;
                LONGXY:units = "degrees_east" ;
                LONGXY:long_name = "longitude" ;
                LONGXY:mode = "time-invariant" ;
        float LATIXY(lat, lon) ;
                LATIXY:units = "degrees_north" ;
                LATIXY:long_name = "latitude" ;
                LATIXY:mode = "time-invariant" ;
        float FSDS(time, lat, lon) ;
                FSDS:_FillValue = 1.e+36f ;
                FSDS:units = "W/m**2" ;
                FSDS:long_name = "incident shortwave radiation" ;
                FSDS:mode = "time-dependent" ;
                FSDS:missing_value = 1.e+36f ;

and one of the blended files:

        float lon(lon) ;
                lon:_FillValue = NaNf ;
        float lat(lat) ;
                lat:_FillValue = NaNf ;
        float LONGXY(lat, lon) ;
                LONGXY:_FillValue = NaNf ;
                LONGXY:units = "degrees_east" ;
                LONGXY:long_name = "longitude" ;
                LONGXY:mode = "time-invariant" ;
        float LATIXY(lat, lon) ;
                LATIXY:_FillValue = NaNf ;
                LATIXY:units = "degrees_north" ;
                LATIXY:long_name = "latitude" ;
                LATIXY:mode = "time-invariant" ;
        float FSDS(lat, time, lon) ;
                FSDS:_FillValue = NaNf ;

For now I will switch my attention back to ctsm5.3 tests as we discussed.

[wwieder]

ah, that seems like an issue.

[slevis-lmwg]

By the way... in case it matters... it probably doesn't at this point:
I accidentally deleted clmforc.GSWP3.c2011.0.5x0.5.Solr.1920-12.nc
in my izumi directory /project/tss/slevis/atm_forcing.datm7.GSWP3.0.5d.v1.c200929

[slevis-lmwg]

List of nco commands that make a blended file more like a CRUJRA file. First I will test 1901 to see if the run will work.

Summary:
Delete _FillValue from most variables.
Update time attributes.
Update lon attributes.
Update lat attributes.
Update FSDS attributes.
Order time before lat dimension.

ncatted -O -a _FillValue,time,d,, clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 
ncatted -O -a _FillValue,lon,d,, clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 
ncatted -O -a _FillValue,lat,d,, clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 
ncatted -O -a _FillValue,LONGXY,d,, clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 
ncatted -O -a _FillValue,LATIXY,d,, clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 

ncatted -a units,time,m,c,'days since 1901-01-01 00:00:00' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc
ncatted -O -a long_name,time,c,sng,'observation time' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc

ncatted -O -a units,lon,c,sng,'degrees_east' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 
ncatted -O -a long_name,lon,c,sng,'longitude' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 
ncatted -O -a mode,lon,c,sng,'time-invariant' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 

ncatted -O -a units,lat,c,sng,'degrees_north' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 
ncatted -O -a long_name,lat,c,sng,'latitude' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 
ncatted -O -a mode,lat,c,sng,'time-invariant' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 

ncatted -a _FillValue,FSDS,m,float,1.e+36 clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 
ncatted -O -a units,FSDS,c,sng,'W/m**2' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 
ncatted -O -a long_name,FSDS,c,sng,'incident shortwave radiation' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 
ncatted -O -a mode,FSDS,c,sng,'time-dependent' clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc 
ncatted -O -a missing_value,FSDS,c,float,1.e+36 clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc

ncpdq -a time,lat clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901.nc clmforc.CRUJRAv2.5_0.5x0.5.Solr.1901_new.nc

I made scripts that execute the above for Solr, Prec, and TPQWL automatically. If all this works, then I can modify the scripts to operate on all the years rather than just 1901.

[slevis-lmwg]

Good news. The above updates to the 1901 files made a difference, and the new simulation is running now. I will wait until we look at results before I also update 1902-1920.

Meanwhile the hourly output is on izumi here (except the third simulation is in progress):

/scratch/cluster/slevis/archive/ctsm52028_f19_GSWP3
/scratch/cluster/slevis/archive/ctsm52028_f19_TR24blended
/scratch/cluster/slevis/archive/ctsm52028_f19_TR24bl_actual

The second simulation should NOT say "blended" in its name (sorry).

[wwieder]

Thanks Sam, I'll correct this in the notebook when we make the full dataset

[slevis-lmwg]

@olyson and I were troubleshooting (in the last hour) a new issue that I came across with the blended data, and we discovered that the mesh file has an additional longitude. In particular, both 360 and zero appear, instead of just 0, and the file has a larger node count than expected as result.

I have the following hypothesis at the moment:
I generated the mesh file from one of the datm files before "fixing" them. One of the "fixes" was a reorder of the (time, lat, lon) dimensions. I will regenerate the mesh file with a "fixed" datm file to see if it resolves the issue.

[slevis-lmwg]

Previous hypothesis: wrong.

However, good news:
Seems I misspoke at this morning's meeting when I said that all 3 methods by which I generated mesh files resulted in identical files. From what I see now, the mesh_maker tool added an unnecessary longitude (360) in the mesh file that the other methods did not. It didn't crash the model but changed answers in the first column (0-degree meridian).

I just looked at a new simulation using one of my other mesh files and confirmed this.

@olyson you can now start the spin-up: The new mesh file has the same name that I told you earlier, but you can see that it doesn't have the extra longitude at 360.

[olyson]

As @slevis-lmwg and I discussed, I ran two cases with hourly output on Derecho to verify the mesh file. One with the original atm forcing and mesh file (/glade/derecho/scratch/oleson/temp_work/TRENDY2024/inputs/three_stream, copied from /project/tss/slevis/TRENDY2024/inputs/three_stream/) and the other with the blended atm forcing and new mesh file (/glade/derecho/scratch/slevis/temp_work/TRENDY2024/inputs/three_stream). The forcing was bfb except over Antarctica.

@slevis-lmwg , the two cases are here:
/glade/work/oleson/ctsm5.3.n04_ctsm5.2.028/cime/scripts/ctsm53n04ctsm52028_f09_blendedhourly_AD
/glade/work/oleson/ctsm5.3.n04_ctsm5.2.028/cime/scripts/ctsm53n04ctsm52028_f09_orighourly_AD
The diferenced hourly history file is here:
/glade/derecho/scratch/oleson/ANALYSIS/blendedhourly-orighourly.nc

So I've started the ne30 and f09 spinups with the blended atm forcing and mesh file.

[ekluzek]

@adamrher in the table above linked here...

#2403 (comment)

There are finidat files for f19 for 2011 and 2003 as well as 2013 for the CONUS grid. I want to simplify our initial conditions so we are just supporting 1850, 1979 and 2000. Since 2013 is specific for the CONUS grid it sounds like that is important to have for the CONUS grid alone. But, I wanted to make sure you see it as important to have.

CAM does start simulations for a variety of dates outside of 1850, 1979, 2000 (and 2013) including: 1950, 1969, 1974, 1980, 1995, 1997, 1999, 2004, 2005, 2006, 2010, and 2015. We can't sensibly support all of those, but if we have IC files within about 15 years of the start date that seems pretty reasonable. The only outlier would be 1950 which is only used for the FWHIST compsets and I assume WACCM simulations aren't going to be as sensitive to the initial conditions from the land model.

How does that sound? Is there anyone else you'll want to weigh in on this?

[olyson]

Candidate initial files from latest spinups and historicals:

ne30:
/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm53n04ctsm52028_ne30pg3t232_pSASU.clm2.r.0121-01-01-00000.nc
/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm53n04ctsm52028_ne30pg3t232_hist.clm2.r.1979-01-01-00000.nc
/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm53n04ctsm52028_ne30pg3t232_hist.clm2.r.2000-01-01-00000.nc
f09:
/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm53n04ctsm52028_f09_pSASU.clm2.r.0161-01-01-00000.nc
/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm53n04ctsm52028_f09_hist.clm2.r.1979-01-01-00000.nc
/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm53n04ctsm52028_f09_hist.clm2.r.2000-01-01-00000.nc

[slevis-lmwg]

@olyson thank you for these.

Question:
Is there a similar list of actual f19 files that I could use. If not, I will point to the f09 files for f19 cases.

[olyson]

There is an 1850 f19 file from the PPE spinup (NCAR/LMWG_dev#70) here:
/glade/campaign/cgd/tss/people/oleson/CLM5_restarts/ctsm530_f19_PPE_pSASU.clm2.r.0161-01-01-00000.nc

My understanding based on the last post of this issue is that the PPE group is testing this file now. I haven't run a historical yet which would provide year 1979 and year 2000 initial files.

[wwieder]

I think we're just expecting to provide 1850 ICs for the f19 grid at this point, Keith. Thanks for posting its location here.

[slevis-lmwg]

I'm preparing to merge #2821, and I'm leaving this issue open, in case there are loose ends to tie here.

[wwieder]

We'll get to do this again (and again) over the next few months. I feel like we should close this for now? Will put this as next to see what others think?