review units in AI data

[cjyetman]

#> Error in base::tryCatch(base::withCallingHandlers({: 4 assertions failed:
#>  * Variable 'data$technology': must contain only valid technology
#>  * names, but has additional element "ICE Hydrogen_HDV".
#>  * Variable 'data$ald_production_unit': must contain only valid
#>  * production units, but has additional elements "# vehicles", "dwt
#>  * km", "t cement", "t coal", and "t steel".
#>  * Variable 'data$ald_emissions_factor_unit': must contain only valid
#>  * emissions factor units, but has additional elements "tCO2/dwt km",
#>  * "tCO2/km", "tCO2/pkm", "tCO2/tkm", "tCO2e/GJ", "tCO2e/MWh", "tCO2e/t
#>  * cement", "tCO2e/t coal", and "tCO2e/t steel".
#>  * Variable 'data$technology': must contain only valid technology names
#>  * for HDV, but has additional elements %s.

This part is pretty important. Could be coming from two places, either:

• AI decided to update the values it uses for emission factors and what not (e.g. tCO2/ km -> tCO2/km)
• AI is providing certain new data points in the output that are not even expected (e.g. tCO2/dwt km)
• (or both?)

I guess we have a decision to make regarding:

• Updating pacta.scenario.preparation and pacta.data.validation to expect what is here
• Updating masterdata_debt to match units expected by pacta.scenario.preparation and pacta.data.validation

Of the two, I guess I would probably prefer the former, which will require a PR to:
https://github.com/RMI-PACTA/pacta.scenario.preparation
But happy to do either, thoughts @cjyetman @AlexAxthelm ?

Hopefully, we just need to adjust the strings themselves, and don't need to change the actual data/ assumptions at all.

Originally posted by @jdhoffa in #185 (comment)

root_dir <- "~/data/workflow-data-preparation-outputs/2023Q4_20240303T082642Z"

pacta.data.validation::validate_masterdata_debt_datastore(
  readRDS(file.path(root_dir, "masterdata_debt_datastore.rds"))
)
#> Error in base::tryCatch(base::withCallingHandlers({: 4 assertions failed:
#>  * Variable 'data$technology': must contain only valid technology
#>  * names, but has additional element "ICE Hydrogen_HDV".
#>  * Variable 'data$ald_production_unit': must contain only valid
#>  * production units, but has additional elements "# vehicles", "dwt
#>  * km", "t cement", "t coal", and "t steel".
#>  * Variable 'data$ald_emissions_factor_unit': must contain only valid
#>  * emissions factor units, but has additional elements "tCO2/dwt km",
#>  * "tCO2/km", "tCO2/pkm", "tCO2/tkm", "tCO2e/GJ", "tCO2e/MWh", "tCO2e/t
#>  * cement", "tCO2e/t coal", and "tCO2e/t steel".
#>  * Variable 'data$technology': must contain only valid technology names
#>  * for HDV, but has additional elements %s.

pacta.data.validation::validate_masterdata_ownership_datastore(
  readRDS(file.path(root_dir, "masterdata_ownership_datastore.rds"))
)
#> Error in base::tryCatch(base::withCallingHandlers({: 4 assertions failed:
#>  * Variable 'data$technology': must contain only valid technology
#>  * names, but has additional element "ICE Hydrogen_HDV".
#>  * Variable 'data$ald_production_unit': must contain only valid
#>  * production units, but has additional elements "# vehicles", "dwt
#>  * km", "t cement", "t coal", and "t steel".
#>  * Variable 'data$ald_emissions_factor_unit': must contain only valid
#>  * emissions factor units, but has additional elements "tCO2/dwt km",
#>  * "tCO2/km", "tCO2/pkm", "tCO2/tkm", "tCO2e/GJ", "tCO2e/MWh", "tCO2e/t
#>  * cement", "tCO2e/t coal", and "tCO2e/t steel".
#>  * Variable 'data$technology': must contain only valid technology names
#>  * for HDV, but has additional elements %s.

• related implement pacta.data.validation #185

AB#10380

[cjyetman]

I think this is resolved by

• fix aviation unit in GECO 2022: "gCO2/pkm" -> "tCO2/pkm" pacta.scenario.data.preparation#31

@jdhoffa?

[jdhoffa]

Before closing this, we should wait until:

• we do a fresh re-run scenarios using workflow.scenario.preparation
• and a fresh run of workflow.data.preparation

to ensure that the masterdata_* output now has acceptable units.

But seems like we're probably close!

[cjyetman]

resolved by:

• include recent spellings of production units in AI data pacta.data.validation#66
• add new emissions factor unit spellings in AI data pacta.data.validation#67