Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New Use Case: construct use case verifying GFS cloud forecasts vs. GFS cloud analyses #2743

Open
33 tasks
DanielAdriaansen opened this issue Oct 24, 2024 · 6 comments · May be fixed by #2850
Open
33 tasks

New Use Case: construct use case verifying GFS cloud forecasts vs. GFS cloud analyses #2743

DanielAdriaansen opened this issue Oct 24, 2024 · 6 comments · May be fixed by #2850
Assignees
@CPKalb @j-opatz
Labels
METplus: Clouds reporting: NRL METplus Naval Research Laboratory METplus Project requestor: Navy/NRL Naval Research Laboratory type: new use case Add a new use case
Milestone
METplus-6.1.0

Comments

@DanielAdriaansen
Copy link
Contributor

DanielAdriaansen commented Oct 24, 2024 •
edited
Loading

Describe the New Use Case

This use case will demonstrate verifying forecasts of cloud information using the GFS global 0.25 degree grid using GridStat.

The fields to verify will be cloud fields we identify in GFS output such as:

  • Cloud Fraction
  • Cloud Base Height
  • Cloud Top Height
  • [potentially] Cloud Layers

We will need to search for and identify the proper field names and levels in the GFS files.

The measures of skill that should be included are:

  • Gilbert Skill Score
  • Equitable Threat Score
  • Fraction Skill Score
  • False Alarm Rate
  • Hit Rate
  • Bias

The end goal is for the user to be able to substitute the GFS analysis which we will use as "truth", with a separate GFS-based AI/ML cloud forecast product on the same GFS 0.25 degree grid. This framework is to support them to be able to do this. We may get some sample data of their truth, however due to restrictions on releasing the data we may need to leave the GFS analysis in place.

The user would also like to be able to stratify forecast performance based on categories of cloud types. These cloud types will be provided later on, but we should brainstorm another type of stratification we can perform using an external classification (maybe weather regimes? precipitation type?), or, implement some simple post-processing, for example stratify performance by all clouds e.g. >= 8 km ("high clouds").

Checklist to get working:

Use Case Name and Category

`model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracBaseTop

Input Data

GFS 0.25 degree forecasts and analyses.
List input data types and sources.
Provide a total input file size, keeping necessary data to a minimum.

Acceptance Testing

Describe tests required for new functionality.
As use case develops, provide a run time here

Time Estimate

Estimate the amount of work required here.
Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the new feature down into sub-issues.

  • Add a checkbox for each sub-issue here.

Relevant Deadlines

Must be completed by 12/31/2024

Funding Source

7730022

Define the Metadata

Assignee

  • Select engineer(s) or no engineer required
  • Select scientist(s) or no scientist required

Labels

  • Review default alert labels
  • Select component(s)
  • Select priority
  • Select requestor(s)
  • Select privacy

Milestone and Projects

  • Select Milestone as a METplus-Wrappers-X.Y.Z version, Consider for Next Release, or Backlog of Development Ideas
  • For a METplus-Wrappers-X.Y.Z version, select the METplus-Wrappers-X.Y.Z Development project

Define Related Issue(s)

Consider the impact to the other METplus components.

New Use Case Checklist

See the METplus Workflow for details.

  • Complete the issue definition above, including the Time Estimate and Funding source.
  • Fork this repository or create a branch of develop.
    Branch name: feature_<Issue Number>_<Description>
  • Complete the development and test your changes.
  • Add/update log messages for easier debugging.
  • Add/update unit tests.
  • Add/update documentation.
  • Add any new Python packages to the METplus Components Python Requirements table.
  • For any new datasets, an entry to the METplus Verification Datasets Guide.
  • Push local changes to GitHub.
  • Submit a pull request to merge into develop.
    Pull request: feature <Issue Number> <Description>
  • Define the pull request metadata, as permissions allow.
    Select: Reviewer(s) and Development issue
    Select: Milestone as the next official version
    Select: METplus-Wrappers-X.Y.Z Development project for development toward the next official release
  • Iterate until the reviewer(s) accept your changes. Merge branch into develop.
  • Create a second pull request to merge develop into develop-ref, following the same steps for the first pull request.
  • Delete your fork or branch.
  • Close this issue.
@DanielAdriaansen DanielAdriaansen added requestor: Navy/NRL Naval Research Laboratory type: new use case Add a new use case reporting: NRL METplus Naval Research Laboratory METplus Project labels Oct 24, 2024
@DanielAdriaansen DanielAdriaansen added this to the METplus-6.1.0 milestone Oct 24, 2024
@DanielAdriaansen DanielAdriaansen moved this from 🟢 Ready to 📖 Backlog in METplus-Wrappers-6.1.0 Development Oct 24, 2024
@DanielAdriaansen DanielAdriaansen moved this from 📖 Backlog to 🏗 In progress in METplus-Wrappers-6.1.0 Development Oct 24, 2024
@DanielAdriaansen DanielAdriaansen moved this from 🏗 In progress to 🟢 Ready in METplus-Wrappers-6.1.0 Development Oct 24, 2024
@DanielAdriaansen DanielAdriaansen changed the title New Use Case: construct use case verifying GFS cloud forecast vs. GFS cloud analyses New Use Case: construct use case verifying GFS cloud forecasts vs. GFS cloud analyses Oct 24, 2024
This was referenced Oct 24, 2024
Open
Open
@DanielAdriaansen
Copy link
Contributor Author

GFS data are here:
/d1/projects/METplus/METplus_Data/development/nrl/cloud/GFS_0.25

Using this command:

wgrib2 -v gfs.0p25.2024030700.f012.grib2 | grep cloud

I see relevant variables:

630:450488137:d=2024030700:LCDC Low Cloud Cover [%]:low cloud layer:12 hour fcst:
631:451333325:d=2024030700:LCDC Low Cloud Cover [%]:low cloud layer:6-12 hour ave fcst:
632:452258290:d=2024030700:MCDC Medium Cloud Cover [%]:middle cloud layer:12 hour fcst:
633:452866413:d=2024030700:MCDC Medium Cloud Cover [%]:middle cloud layer:6-12 hour ave fcst:
634:453577593:d=2024030700:HCDC High Cloud Cover [%]:high cloud layer:12 hour fcst:
635:454309341:d=2024030700:HCDC High Cloud Cover [%]:high cloud layer:6-12 hour ave fcst:
638:457005258:d=2024030700:HGT Geopotential Height [gpm]:cloud ceiling:12 hour fcst:
639:458203404:d=2024030700:PRES Pressure [Pa]:convective cloud bottom level:12 hour fcst:
640:458750807:d=2024030700:PRES Pressure [Pa]:low cloud bottom level:6-12 hour ave fcst:
641:460188975:d=2024030700:PRES Pressure [Pa]:middle cloud bottom level:6-12 hour ave fcst:
642:461414641:d=2024030700:PRES Pressure [Pa]:high cloud bottom level:6-12 hour ave fcst:
643:462935144:d=2024030700:PRES Pressure [Pa]:convective cloud top level:12 hour fcst:
644:463536141:d=2024030700:PRES Pressure [Pa]:low cloud top level:6-12 hour ave fcst:
645:464992169:d=2024030700:PRES Pressure [Pa]:middle cloud top level:6-12 hour ave fcst:
646:466180239:d=2024030700:PRES Pressure [Pa]:high cloud top level:6-12 hour ave fcst:
647:467707298:d=2024030700:TMP Temperature [K]:low cloud top level:6-12 hour ave fcst:
648:468725963:d=2024030700:TMP Temperature [K]:middle cloud top level:6-12 hour ave fcst:
649:469628288:d=2024030700:TMP Temperature [K]:high cloud top level:6-12 hour ave fcst:
650:470836489:d=2024030700:TCDC Total Cloud Cover [%]:convective cloud layer:12 hour fcst:
651:471552157:d=2024030700:TCDC Total Cloud Cover [%]:boundary layer cloud layer:6-12 hour ave fcst:

In addition, there is an isobaric "total cloud cover" variables (TCDC) at various pressure levels in the file.

@j-opatz
Copy link
Contributor

j-opatz commented Nov 5, 2024

@CPKalb you and I may need to tackle this one; A GFS-to-GFS gridded comparison seems...too basic. We have plenty of examples where this type of verification is completed, including some of the cloud verification use cases (some of which are more complex and still adaptable).

I think in the last NRL meeting we discussed looping in a MODE analysis/DMAP approach on this; I'd like for both of us to do some research and see if there's any cloud-identification research that could shed a light on a path forward. I'll try to find some time to meet up on Fri-early Mon to discuss our findings and settle on an approach.

@CPKalb
Copy link
Contributor

CPKalb commented Nov 5, 2024 •
edited
Loading

We have a use case that uses distance maps on brightness temperatures. I can't remember if there was a paper or anything associated with this though. I'll have to poke around.
https://metplus.readthedocs.io/en/latest/generated/model_applications/short_range/GridStat_fcstFV3_obsGOES_BrightnessTempDmap.html#sphx-glr-generated-model-applications-short-range-gridstat-fcstfv3-obsgoes-brightnesstempdmap-py

https://metplus.readthedocs.io/en/latest/generated/model_applications/short_range/MODE_fcstFV3_obsGOES_BrightnessTempObjs.html#sphx-glr-generated-model-applications-short-range-mode-fcstfv3-obsgoes-brightnesstempobjs-py

@JohnHalleyGotway JohnHalleyGotway moved this from 🟢 Ready to 🏗 In progress in METplus-Wrappers-6.1.0 Development Nov 6, 2024
@j-opatz
Copy link
Contributor

j-opatz commented Nov 11, 2024

Adding notes here so direction of use case is recorded:

It sounds like (for the time being) we should just focus on a cloud fraction comparison between the two and exhibit the difference between traditional statistics and an object based approach. So the use case would be a GridStat call and a MODE call, looking at 1 (or more) of the fractional cloud coverage heights (low, mid, high, or convective).

@CPKalb
Copy link
Contributor

CPKalb commented Nov 13, 2024

Is there a date I should start with for this verification or should I just pick something?

@DanielAdriaansen
Copy link
Contributor Author

Is there a date I should start with for this verification or should I just pick something?

Dates are discussed here: https://docs.google.com/document/d/1SJb2ZBrLG9jqH_Tu_ssB8lqeDxnPx1Xp-FHFRdj7HM0/edit?tab=t.0#heading=h.76lq03tjjtx6

3/7/2024 00 UTC initialization with forecasts leads of 6,12,18h and valid times of 06, 12, and 18 UTC on 3/7/2024.

@CPKalb CPKalb linked a pull request Dec 24, 2024 that will close this issue
Feature 2743 nrl gfs cloud analysis #2850
Open
16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@CPKalb CPKalb

@j-opatz j-opatz

Labels
METplus: Clouds reporting: NRL METplus Naval Research Laboratory METplus Project requestor: Navy/NRL Naval Research Laboratory type: new use case Add a new use case
Projects
METplus-Wrappers-6.1.0 Development
Status: 🏗 In progress
Milestone
METplus-6.1.0
Development

Successfully merging a pull request may close this issue.

Feature 2743 nrl gfs cloud analysis dtcenter/METplus
3 participants
@DanielAdriaansen @CPKalb @j-opatz