Charter 23 : Manual testing of interpolating surface temperature and pressure values from CAMS data to use in converting OpenAQ ppm to µg m³

Testing ticket Convert PPM measurements from OpenAQ data based on CAMS air density.

Description

• We have an automated test that verifies the values µg/m³ generated from ppm values using a specified estimated_surface_pressure_pa and estimated_temperature_k
• A manual verification is needed to ensure that the estimated_surface_pressure_pa and estimated_temperature_k are correct. They are generated by interpolating data from a given GRIB file, for a given latitude, longitude and time.

Test Process - test executed on 02/07/2024

• Run the test__convert_ppm_to_ugm3_and_store__only_no2_so2_o3 test in the open_aq_etl_tests.py
  • This seeds the in_situ database with a document that ppm values for pollutants
  • latitude_berlin = 52.52437, longitude_berlin = 13.41053 and date_utc = "2024-06-26T13:10:20+00:00"
  • the test uses GRIB data from 2024-06-26_12
• Manually open the single-level and multi-level gribs for 2024-06-26_12 using xr.open_dataset().
• Convert each dataset to dataframes, and then to csv files
• Filter the data in each csv by latitude and longitude and time to isolate the nearest values to the specified latitude, longitude and time
• Create a dataset using this data

time = pd.date_range(start="2024-06-26T12:00", end="2024-06-26T15:00", periods=2)
latitude = [52.4, 52.8]
longitude = [13.2, 13.6]

sp_data = np.array(
    [
        [[100615.58, 100467.2], [100665.58, 100518.2]],
        [[100717.58, 100573.2], [100667.58, 100521.2]],
    ]
)


t_data = np.array(
    [
        [[301.6565, 302.48303], [301.52954, 302.23303]],
        [[300.83423, 301.23108], [300.50415, 300.93225]],
    ]
)


ds = xr.Dataset(
    {
        "sp": (["latitude", "longitude", "time"], sp_data),
        "t": (["latitude", "longitude", "time"], t_data),
    },
    {
        "latitude": latitude,
        "longitude": longitude,
        "time": time,
    },
)

• export the dataset to csv again to ensure this is identical to the original dataset
• interpolate the data based on the target latitude, longitude and time for the Berlin document

interpolated_data = ds.interp(
    latitude=52.52437,
    longitude=13.41053,
    time="2024-06-26T13:10:20+00:00",
    method="linear",
)

• print the interpolated data.to_dict()

• compare pressure and temperature to that stored in the database

Conclusion

• the databse values match the manual interpolation for this data to at least 4 decimal places