Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update the bufr2ioda yamls #183

Open
wants to merge 9 commits into
base: develop
Choose a base branch
from
Open

Update the bufr2ioda yamls #183

wants to merge 9 commits into from

Conversation

delippi
Copy link
Collaborator

@delippi delippi commented Sep 26, 2024 •
edited by guoqing-noaa
Loading

This PR updates the bufr2ioda yaml configurations to address issues identified in the original files. Some of those issues include:

change height data type from int to float

  • change unit and/or longName to more accurately reflect the data
  • separate multiple subset types (e.g., aircft, aircar, adpsfc, sfcshp, etc.) and create single ioda file for each subset type.
  • separate ObsError/wind to ObsError/windEastward and ObsError/windNorthward
  • using stationPressure instead of pressure for when referring to surface pressure
  • clean up extra white space
  • re-organize to make comparisons easier

@delippi
Initial commit of updated yamls. Includes several changes including s…
724997b 
…ome data types from int to float as expected by JEDI, some units and/or longName changes, separated aircft and aircar, separated adpsfc and sfcshp, some clean up of extra white space, and some re-organization to make comparisons between each yaml easier.
ShunLiu-NOAA
ShunLiu-NOAA previously approved these changes Sep 27, 2024
View reviewed changes
@ShunLiu-NOAA
Copy link

Opps, it is a draft PR...

@delippi delippi marked this pull request as ready for review September 30, 2024 16:37
@delippi delippi self-assigned this Sep 30, 2024
ShunLiu-NOAA
ShunLiu-NOAA reviewed Oct 4, 2024
View reviewed changes
Copy link

@ShunLiu-NOAA ShunLiu-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

ShunLiu-NOAA
ShunLiu-NOAA previously approved these changes Oct 4, 2024
View reviewed changes
guoqing-noaa
guoqing-noaa previously approved these changes Oct 4, 2024
View reviewed changes
ShunLiu-NOAA
ShunLiu-NOAA previously approved these changes Oct 4, 2024
View reviewed changes
@delippi delippi dismissed stale reviews from ShunLiu-NOAA and guoqing-noaa via 1c34239 October 4, 2024 17:50
@spanNOAA
Copy link
Collaborator

spanNOAA commented Oct 4, 2024

After reviewing the YAML files I'm currently using for my FV3JEDI case, I've noticed inconsistencies in the data types for station elevation and height across different files. I usually use integers for surface observations and floats for others. I'm not sure why this is the case, as these settings appear to be derived from the previous YAML examples provided by JCSDA. I will look into this issue further.

@delippi
Copy link
Collaborator Author

delippi commented Oct 17, 2024

After reviewing the YAML files I'm currently using for my FV3JEDI case, I've noticed inconsistencies in the data types for station elevation and height across different files. I usually use integers for surface observations and floats for others. I'm not sure why this is the case, as these settings appear to be derived from the previous YAML examples provided by JCSDA. I will look into this issue further.

@spanNOAA @guoqing-noaa @ShunLiu-NOAA @hu5970 @HuiLiu-NOAA, I know Sijie was going to look into the inconsistencies noted above. But I've reverted heights/stationElevation to integers and don't seem to have an issue at the moment--I can't reproduce the issue I thought I had before. If it comes up we can always just make that change later. Is there anything else anyone would like to see different? Otherwise, I think this PR is ready to go.

@delippi
Copy link
Collaborator Author

delippi commented Oct 17, 2024

Perhaps one thing could be a bash script that can help generate the ioda files using the yamls? Would that be something to add here?

ShunLiu-NOAA
ShunLiu-NOAA previously approved these changes Oct 17, 2024
View reviewed changes
Copy link

@ShunLiu-NOAA ShunLiu-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.
Let's consider adding a bufr2ioda ctest in another PR with including a bash script to generate the IODA files.

@delippi delippi mentioned this pull request Oct 22, 2024
Merged
@spanNOAA
Copy link
Collaborator

@delippi Have you ever looked at the bufr2ioda outputs? When I review my adpsfc output, I consistently find two values for each variable at the same location and time offset. Do you have any insights on this? Here’s a snippet of the obs:

latitude longitude time offset pressure airTemperature
141 24.030001 255.330002 -299.988007 81410.0 NaN
142 24.030001 255.330002 -299.988007 81410.0 290.549988
168 24.549999 278.250000 -1559.988037 101180.0 NaN
169 24.549999 278.250000 -1559.988037 101180.0 301.449982
170 24.549999 278.250000 -839.987976 101210.0 NaN
171 24.549999 278.250000 -839.987976 101210.0 301.449982
172 24.549999 278.250000 -420.011993 101210.0 NaN
173 24.549999 278.250000 -420.011993 101210.0 301.449982
10060 24.556110 278.240570 -420.011993 101150.0 NaN
10061 24.556110 278.240570 -420.011993 101150.0 301.449982
174 24.570000 278.320007 -420.011993 101200.0 NaN
175 24.570000 278.320007 -420.011993 101200.0 299.850006
159 24.629999 252.570007 -299.988007 100870.0 NaN
160 24.629999 252.570007 -299.988007 100870.0 293.149994
176 24.730000 278.950012 -420.011993 101170.0 NaN
177 24.730000 278.950012 -420.011993 101170.0 301.449982
178 25.049999 282.529999 0.000000 101210.0 NaN
179 25.049999 282.529999 0.000000 101210.0 301.149994
180 25.650000 279.570007 -420.011993 101180.0 NaN
181 25.650000 279.570007 -420.011993 101180.0 299.850006

@delippi
Copy link
Collaborator Author

delippi commented Oct 22, 2024

@delippi Have you ever looked at the bufr2ioda outputs? When I review my adpsfc output, I consistently find two values for each variable at the same location and time offset. Do you have any insights on this? Here’s a snippet of the obs:

latitude longitude time offset pressure airTemperature
141 24.030001 255.330002 -299.988007 81410.0 NaN
142 24.030001 255.330002 -299.988007 81410.0 290.549988
168 24.549999 278.250000 -1559.988037 101180.0 NaN
169 24.549999 278.250000 -1559.988037 101180.0 301.449982
170 24.549999 278.250000 -839.987976 101210.0 NaN
171 24.549999 278.250000 -839.987976 101210.0 301.449982
172 24.549999 278.250000 -420.011993 101210.0 NaN
173 24.549999 278.250000 -420.011993 101210.0 301.449982
10060 24.556110 278.240570 -420.011993 101150.0 NaN
10061 24.556110 278.240570 -420.011993 101150.0 301.449982
174 24.570000 278.320007 -420.011993 101200.0 NaN
175 24.570000 278.320007 -420.011993 101200.0 299.850006
159 24.629999 252.570007 -299.988007 100870.0 NaN
160 24.629999 252.570007 -299.988007 100870.0 293.149994
176 24.730000 278.950012 -420.011993 101170.0 NaN
177 24.730000 278.950012 -420.011993 101170.0 301.449982
178 25.049999 282.529999 0.000000 101210.0 NaN
179 25.049999 282.529999 0.000000 101210.0 301.149994
180 25.650000 279.570007 -420.011993 101180.0 NaN
181 25.650000 279.570007 -420.011993 101180.0 299.850006

@spanNOAA, I think that might be because wind is reported separately from the other variables. Check the corresponding ObsType with the NaN's. I bet they start with a 2. Here is a snip from the ioda_adpsfc_dc.nc:

group: ObsType {
  data:
   airTemperature = 287, 187, 287, 187, 287, 187, 287, 187, 287, 187, 287, 


group: ObsValue {
  data:
   airTemperature = _, 303.15, _, 302.15, _, 302.15, _, 302.15, _, 302.15, _,

@delippi delippi mentioned this pull request Oct 22, 2024
Open
guoqing-noaa and others added 2 commits October 23, 2024 07:19
@guoqing-noaa
Merge branch 'develop' into feature/update_prepbufr2ioda_yamls
1ca44e8
@delippi
Updates to match IODA conventions. Changed units for timeOffset to s …
237b6ad 
…and added the conversion from hours, changed mesonetProvider to dataProviderOrigin, added dataProviderSubOrigin for mesonet as well, made various updates to unit labels to match the IODA convention.
@delippi
Copy link
Collaborator Author

delippi commented Oct 23, 2024

I'm trying to follow this https://github.com/JCSDA-internal/ioda/blob/develop/share/ioda/yaml/validation/ObsSpace.yaml (which Praveen provided).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@guoqing-noaa guoqing-noaa guoqing-noaa left review comments

@ShunLiu-NOAA ShunLiu-NOAA ShunLiu-NOAA left review comments

@hu5970 hu5970 Awaiting requested review from hu5970

@HuiLiu-NOAA HuiLiu-NOAA Awaiting requested review from HuiLiu-NOAA

@spanNOAA spanNOAA Awaiting requested review from spanNOAA

At least 1 approving review is required to merge this pull request.

Assignees

@delippi delippi

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@delippi @ShunLiu-NOAA @spanNOAA @guoqing-noaa @HuiLiu-NOAA