Missing dp for ideal simulation

[NikiJacobs]

Hi,

I am new to X-STILT and I thought I had all the necessary inputs for the ideal column simulation and footprint generation. I am getting the following output:

Running simulation ID: 202409121900_-79.1876_43.7843_X
before_trajec_xstilt(): extracting meteo vars from meteo fields...
[1] "2024-09-12 19:00:00 UTC"
get.met.vars(): found meteo files of /home/njacobs/X-STILT/met_data/hrrr/20240912_18-23_hrrr

END of before_trajec_xstilt(), start calculating X-trajectories...
before_footprint_xstilt(): footprint weighting assuming AK = 1 (no satellite dependencies)
Error in mutate(., xdry = (1 - q)/g/Mdry * dp * 100, pwf = xdry/xdry.tot) :
ℹ In argument: xdry = (1 - q)/g/Mdry * dp * 100.
Caused by error:
! object 'dp' not found
In addition: Warning message:
There was 1 warning in mutate().
ℹ In argument: sigma = *....
Caused by warning in sqrt():
! NaNs produced

It is looking for dp, but I am trying to run without the dependence on satellite data, so where should it be getting dp from?

I would greatly appreciate some guidance.

Thanks!

[wde0924]

Thanks for reaching out, Niki! I haven't used the ideal column simulations for a while - it seems that dp has not been passed onto output since I split one function into two (and probably forgot about the ideal simulation). This dp is the difference in pressure between the model levels, not the dp from OCO. Let me double-check the script and fix it this week.

Thanks,

[wde0924]

Hi Niki - bug fixed. Try downloading the latest scripts and let me know if it works now. =) - Dien

[NikiJacobs]

I am now getting the following error when I attempt the ideal case:

Error in if (nchar(tmp_timestr) == 8) ff = "%Y%m%d" :
missing value where TRUE/FALSE needed
Calls: download.met.arl

I have already downloaded hrrr met data files for the day before through the day after. I am trying to run for Toronto 20240912.

Also, I am having problems running the slant column footprint and trajectories with EM27 data. I have a ggg public netcdf with EM27 retrievals, when I run it with obs_sensor = 'EM27' I get:
--- Configuring ---
Need to generate trajec...
Need to generate footprint...
Error in create.outwd(timestr, obs_species, obs_sensor, obs_path, obs_fn, :
ONLY TROPOMI, OCO-2, OCO-3, NA are currently implemented, please check @param obs_sensor...
Calls: config_xstilt -> create.outwd
Execution halted

and when I run with obs_sensor = 'TCCON', I get:
--- Configuring ---
Need to generate trajec...
Need to generate footprint...

Output dir - /home/njacobs/X-STILT/XSTILT_output/UTSG/out_2024091200_hrrr_TCCON
Obs file - /home/njacobs/X-STILT/input_data/EM27/GGG2020/UTSG/ta20240912_20240912.experimental.public.qc.nc
[1] "vobjtovarid4: error #F: I could not find the requsted var (or dimvar) in the file!"
[1] "var (or dimvar) name: xco2_x2019"
[1] "file name: /home/njacobs/X-STILT/input_data/EM27/GGG2020/UTSG/ta20240912_20240912.experimental.public.qc.nc"
Error in vobjtovarid4(nc, varid, verbose = verbose, allowdimvar = TRUE) :
Variable not found
Calls: config_xstilt ... get.sensor.obs -> grab_tccon -> ncvar_get -> vobjtovarid4
Execution halted

I guess you are still working on setting it to run with the EM27 data? Though I saw that it was used with EM27 data in Mexico City already.

[wde0924]

Ah, I suppose for my testing ideal runs, my timestr is not NA. Could you share your receptor_demo.csv file as well as the run_xstilt.r you modified for your own receptor locations for EM27?

Yes - I don't have the scripts customized for EM27 yet. For the Mexico City analysis, Ke Che did her own code modifications using X-STILT. And, we have been chatting to merge hers. Could you point me to the ggg public netcdf with EM27 retrieval? Do they share similar data format as TCCON?

For the TCCON, let me check the subroutine for reading TCCON files. I will make this the priority! =)

[wde0924]

Okay, my runs for Lamont TCCON on July 1, 2020, seemed to work well with a particle.rds file generated (with slant column correction in receptor locations) - no bug with grab_tccon. I suspect the experimental tccon file may have a different data format from the official public one?

Essentially the subroutine grab_tccon.r is trying to load TCCON data for the desired time period and grab variables of XCO2 for both X2007 and X2019 WMO standards. But, it could not find a variable named xco2_x2019 in your experimental tccon file, somehow.

Here is my output running TCCON:
--- Configuring ---
Need to generate trajec...

Output dir - /central/groups/POW/XSTILT_output/Lamont/out_2020070120_hrrr_TCCON
Obs file - /central/groups/POW/TCCON/GGG2020/oc20110416_20221022.public.qc.nc
Done with receptor setup...total 17 receptor(s)..

NO horizontal wind error component for generating trajec...
NO PBL scaling when generating trajec...

Initializing X-STILT
Number of receptors: 17
Number of parallel threads: 120
Multi node parallelization using slurm. Dispatching jobs...
Submitted batch job 44618941

[wde0924]

Niki, for the ideal simulation - I fixed the download.met.arl and the main script to let it grab timestr from receptor_demo.csv (latest commit cea1d2e). I also tested out a receptor_demo.csv for random points in Toronto with different days (e.g., https://github.com/uataq/X-STILT/blob/master/receptor_demo.csv), which seems to work on my side. The latest script will download all met files for the multiple days and their backward time. Please try it out again and let me know if it works.

Dien

[NikiJacobs]

Hi Dien, sorry for the slow reply. It seems like the ideal case is running fine. I still get these warnings:
5: There was 1 warning in mutate().
ℹ�In argument: sigma = *....
Caused by warning in sqrt():
!�NaNs produced

It did generate footprints though.
For the runs with EM27 data, they only have an xco2 variable, no xco2_x2007 or xco2_x2019. I may try to edit the code, but will not have time to work on it until Friday. Thanks!

[wde0924]

Hi Niki, you can ignore this warning. It sometimes occurred when met fields especially HRRR have a certain missing variable. For working TCCON, I am using the public qc file with contains xco2_x$YYYY. But we can do a quick change to the subroutine (to bypass reading xCO2 from column obs) since extracting xCO2 from TCCON is not necessary when running trajectories. Let me know what you prefer. - Dien