Challenge 14 - Jupyter notebooks for CAMS web charts

[RubenRT7]

Challenge 14 - Jupyter notebooks for CAMS web charts

Stream 1 - Data Visualization and visual narratives for Earth Sciences applications

Goal

Develop Jupyter notebooks for reproducing CAMS global and European scale forecast charts.

Mentors and skills

• Mentors: Mark Parrington, Miha Razinger, Milana Vuckovic, Chris Stewart (all ECMWF)
• Skills required:
  • Python programming (earthkit-data, earthkit-maps, matplotlib, cartopy)
  • Processing and visualization of scientific data (netCDF, GRIB…)
  • Jupyter notebooks

---

Challenge description

The Copernicus Atmosphere Monitoring Service (CAMS) produces a wide range of forecast charts for atmospheric composition and air quality at the global (1) and European scale (2) every day. Forecast charts are published on the CAMS website but can be difficult to reproduce by users and the media.

All the forecasts are freely and openly available from the Atmosphere Data Store (ADS) (3). Given the open access nature of CAMS, and more generally Copernicus, data there is a strong requirement for users to be able to more effectively access and plot the forecast products. Jupyter notebooks have been developed for ECMWF web charts but have not yet been extended to the CAMS charts.

In addition to the web charts, CAMS also regularly shares information on its forecasts with users through social media, media, and training activities. The development of Jupyter notebooks for CAMS charts will also feed directly into these user outreach and communication activities to enhance the uptake of CAMS products with expert and non-expert users.

The project will be covering the following aspects:

• Develop a standard Jupyter notebook template, or templates, which can be applied in a flexible manner to the web charts currently published on the CAMS website for global and regional forecasts.
• Utilise and test ECMWF earthkit python library for data processing and visualization.
• Test portability for using the Jupyter notebooks in training and communication applications.
• Optional development to map earthkit libraries to other commonly available open source libraries.

Implementation proposal

All the data required to reproduce the plots is hosted on the ADS. We envisage that each Notebook for a web product will be generated programmatically based on a template and a configuration file. The type of information contained in a configuration file:

• mapping between a web product name and a list of specific keys that should be used in an ADS API request (dataset name, variable name, level type ...)
• unit or physical quantity conversion formulae
• graphical attributes required to reproduce a chart like contour colour map and contour intervals (the CAMS website plots are currently generated using the ECMWF Magics plotting library. Your task will be to convert the current configuration to matplotlib code

We welcome proposals using different approaches that would solve the same problems.

A practical example

Let’s try to reproduce a forecast chart which is available at (https://atmosphere.copernicus.eu/charts/packages/cams/products/nitrogen-dioxide-forecasts?layer_name=composition_no2_surface&projection=classical_global):

1. Data request

c.retrieve(
'cams-global-atmospheric-composition-forecasts',
{
'date': '2024-03-13/2024-03-13',
'type': 'forecast',
'format': 'grib',
'variable': 'nitrogen_dioxide',
'model_level': '137',
'time': '00:00',
'leadtime_hour': '12',
},
'download.grib')

2. Unit conversion

To convert the values from mass mixing ratio to volume mixing ratio in ppbv we need to multiply then by:

Factor = 28.97 / 46.0055 * 1e9

3. Translating plot attributes

We will need to convert Magics plot style settings to Python. This is an extract from the current configuration file:

...

"contour_level_list": "0.01/0.02/0.05/0.1/0.2/0.5/1/2/5/10/15/20/50/100/300",

"contour_shade_colour_list": "rgb(0.607,0.780,0.917)/rgb(0.000,0.619,0.878)/rgb(0.000,0.482,0.749)/rgb(0.149,0.356,0.627)/rgb(0.149,0.568,0.439)/rgb(0.419,0.698,0.356)/rgb(0.658,0.788,0.419)/rgb(1.000,0.956,0.607)/rgb(0.956,0.819,0.219)/rgb(0.917,0.709,0.219)/rgb(0.866,0.529,0.250)/rgb(0.800,0.207,0.200)/rgb(0.607,0.086,0.137)/rgb(0.337,0.098,0.098)", ...

Deliverables

We expect two distinct deliverables. One is a Python script that creates a Jupyter Notebook for a given CAMS web chart based on a configuration file. The other is a collection of Notebooks for every chart that we publish daily.

References

https://atmosphere.copernicus.eu/charts/packages/cams 

https://atmosphere.copernicus.eu/charts/packages/cams_air_quality/products/europe-air-quality-forecast-regulated 

https://ads.atmosphere.copernicus.eu/