implement `pacta.data.validation` #185

cjyetman · 2024-03-07T11:08:50Z

closes #18

Most validation errors originally found (below) have been resolved. Validation of financial_data and abcd_flags_equity has been removed for now and will be added in the future.

investigation issues:

relevant fixes in pacta.data.validation:

relevant fix in pacta.data.preparation

filter out factset_entity_id that are NA and therefore not useful pacta.data.preparation#18

validation of financial_data and abcd_flags_equity has been removed from this PR, and future intended implementation is tracked here

~~currently, a few of the validation checks fail using 2023Q4 config~~

root_dir <- "~/data/workflow-data-preparation-outputs/2023Q4_20240303T082642Z"

pacta.data.validation::validate_currencies(
  readRDS(file.path(root_dir, "currencies.rds"))
)

pacta.data.validation::validate_financial_data(
  readRDS(file.path(root_dir, "financial_data.rds"))
)
#> Error in base::tryCatch(base::withCallingHandlers({: 1 assertions failed:
#>  * Variable 'data$isin': must contain only valid ISINs, but has
#>  * additional elements "US78467KA#42", "US85255AA*13", "NOR6778@AB29",
#>  * "US048303E#47", "US313855F*48", "US372460A#28", "CA820439A#42",
#>  * "CA008474E@39", "US941848D#79", "US524908@431", "US00253#AB60",
#>  * "US615369A@49", "US893578A@31", "US57555*AD16", "BEB7227*AB60",
#>  * "US031100A@92", "US628464A#60", "US10468*AE44", …, "US758750B*36",
#>  * and "US15135#BQ49".

pacta.data.validation::validate_masterdata_debt_datastore(
  readRDS(file.path(root_dir, "masterdata_debt_datastore.rds"))
)
#> Error in base::tryCatch(base::withCallingHandlers({: 4 assertions failed:
#>  * Variable 'data$technology': must contain only valid technology
#>  * names, but has additional element "ICE Hydrogen_HDV".
#>  * Variable 'data$ald_production_unit': must contain only valid
#>  * production units, but has additional elements "# vehicles", "dwt
#>  * km", "t cement", "t coal", and "t steel".
#>  * Variable 'data$ald_emissions_factor_unit': must contain only valid
#>  * emissions factor units, but has additional elements "tCO2/dwt km",
#>  * "tCO2/km", "tCO2/pkm", "tCO2/tkm", "tCO2e/GJ", "tCO2e/MWh", "tCO2e/t
#>  * cement", "tCO2e/t coal", and "tCO2e/t steel".
#>  * Variable 'data$technology': must contain only valid technology names
#>  * for HDV, but has additional elements %s.

pacta.data.validation::validate_masterdata_ownership_datastore(
  readRDS(file.path(root_dir, "masterdata_ownership_datastore.rds"))
)
#> Error in base::tryCatch(base::withCallingHandlers({: 4 assertions failed:
#>  * Variable 'data$technology': must contain only valid technology
#>  * names, but has additional element "ICE Hydrogen_HDV".
#>  * Variable 'data$ald_production_unit': must contain only valid
#>  * production units, but has additional elements "# vehicles", "dwt
#>  * km", "t cement", "t coal", and "t steel".
#>  * Variable 'data$ald_emissions_factor_unit': must contain only valid
#>  * emissions factor units, but has additional elements "tCO2/dwt km",
#>  * "tCO2/km", "tCO2/pkm", "tCO2/tkm", "tCO2e/GJ", "tCO2e/MWh", "tCO2e/t
#>  * cement", "tCO2e/t coal", and "tCO2e/t steel".
#>  * Variable 'data$technology': must contain only valid technology names
#>  * for HDV, but has additional elements %s.

pacta.data.validation::validate_abcd_flags_bonds(
  readRDS(file.path(root_dir, "abcd_flags_bonds.rds"))
)
#> Error in assert_valid_factset_entity_id(data[[col_name]], add = coll, : Assertion on 'data$credit_parent_id' failed: must contain only valid FactSet entity IDs, but has additional element NA.

pacta.data.validation::validate_abcd_flags_equity(
  readRDS(file.path(root_dir, "abcd_flags_equity.rds"))
)
#> Error in base::tryCatch(base::withCallingHandlers({: 1 assertions failed:
#>  * Variable 'data$isin': must contain only valid ISINs, but has
#>  * additional elements "US78467KA#42", "US85255AA*13", "NOR6778@AB29",
#>  * "US048303E#47", "US313855F*48", "US372460A#28", "CA820439A#42",
#>  * "CA008474E@39", "US941848D#79", "US524908@431", "US00253#AB60",
#>  * "US615369A@49", "US893578A@31", "US57555*AD16", "BEB7227*AB60",
#>  * "US031100A@92", "US628464A#60", "US10468*AE44", …, "US758750B*36",
#>  * and "US15135#BQ49".

jdhoffa · 2024-03-07T11:33:18Z

💯

jdhoffa · 2024-03-07T14:10:07Z

#> Error in base::tryCatch(base::withCallingHandlers({: 1 assertions failed:
#>  * Variable 'data$isin': must contain only valid ISINs, but has
#>  * additional elements "US78467KA#42", "US85255AA*13", "NOR6778@AB29",
#>  * "US048303E#47", "US313855F*48", "US372460A#28", "CA820439A#42",
#>  * "CA008474E@39", "US941848D#79", "US524908@431", "US00253#AB60",
#>  * "US615369A@49", "US893578A@31", "US57555*AD16", "BEB7227*AB60",
#>  * "US031100A@92", "US628464A#60", "US10468*AE44", …, "US758750B*36",
#>  * and "US15135#BQ49".

AFAIK, those are valid ISINs, they are just ISINs of private securities (venture capital, and things like that). Had a look at this when I was looking into Private equity, it's sorta cool that we have them actually! I would loosen the restrictions on the expectation there personally (but maybe ask Nick for a second opinion?)

But in any case, TM doesn't publicize that it can cover private equity, so I think it's totally fine to just filter them out.

jdhoffa · 2024-03-07T14:18:56Z

#> Error in base::tryCatch(base::withCallingHandlers({: 4 assertions failed:
#>  * Variable 'data$technology': must contain only valid technology
#>  * names, but has additional element "ICE Hydrogen_HDV".
#>  * Variable 'data$ald_production_unit': must contain only valid
#>  * production units, but has additional elements "# vehicles", "dwt
#>  * km", "t cement", "t coal", and "t steel".
#>  * Variable 'data$ald_emissions_factor_unit': must contain only valid
#>  * emissions factor units, but has additional elements "tCO2/dwt km",
#>  * "tCO2/km", "tCO2/pkm", "tCO2/tkm", "tCO2e/GJ", "tCO2e/MWh", "tCO2e/t
#>  * cement", "tCO2e/t coal", and "tCO2e/t steel".
#>  * Variable 'data$technology': must contain only valid technology names
#>  * for HDV, but has additional elements %s.

This part is pretty important. Could be coming from two places, either:

AI decided to update the values it uses for emission factors and what not (e.g. tCO2/ km -> tCO2/km)
AI is providing certain new data points in the output that are not even expected (e.g. tCO2/dwt km)
(or both?)

I guess we have a decision to make regarding:

Updating pacta.scenario.preparation and pacta.data.validation to expect what is here
Updating masterdata_debt to match units expected by pacta.scenario.preparation and pacta.data.validation

Of the two, I guess I would probably prefer the former, which will require a PR to:
https://github.com/RMI-PACTA/pacta.scenario.preparation
But happy to do either, thoughts @cjyetman @AlexAxthelm ?

Hopefully, we just need to adjust the strings themselves, and don't need to change the actual data/ assumptions at all.

jdhoffa · 2024-03-07T14:20:38Z

This seems to just be an issue with HDV, which I don't really use at all (it's still not supported in r2dii.*) so I don't have any useful feedback there.

What technology name were you using for HDV here?

#>  * Variable 'data$technology': must contain only valid technology names
#>  * for HDV, but has additional elements %s.

cjyetman · 2024-04-17T06:55:17Z

relevant fixes in pacta.data.validation:

relevant fix in pacta.data.preparation

filter out factset_entity_id that are NA and therefore not useful pacta.data.preparation#18

validation of financial_data has been removed from this PR, and future intended implementation is tracked here

implement validation for financial_data and abcd_flags_equity when ready #222
dependent on allow technically invalid, but likely intentional ISINs with a warning pacta.data.validation#69

implement pacta.data.validation

1422b45

Merge branch 'main' into use-pacta.data.validation

f69fdd2

Merge branch 'main' into use-pacta.data.validation

93d6b06

cjyetman changed the title ~~implement pacta.data.validation~~ implement pacta.data.validation Mar 14, 2024

cjyetman added 3 commits March 15, 2024 10:39

Merge branch 'main' into use-pacta.data.validation

843345b

Merge branch 'main' into use-pacta.data.validation

01b4ff0

Merge branch 'main' into use-pacta.data.validation

5cbdb67

cjyetman added the priority label Mar 23, 2024

This was referenced Mar 25, 2024

review invalid ISINs in FactSet data #196

Closed

review units in AI data #197

Closed

review HDV technologies in AI data #198

Closed

review invalid entity IDs in FactSet entity_info #199

Closed

cjyetman added 6 commits March 25, 2024 13:24

fix syntax

5af85dd

Merge branch 'main' into use-pacta.data.validation

4922fe2

Merge branch 'main' into use-pacta.data.validation

348901b

Merge branch 'main' into use-pacta.data.validation

322727c

Merge branch 'main' into use-pacta.data.validation

b3d13d9

Merge branch 'main' into use-pacta.data.validation

1612861

cjyetman marked this pull request as ready for review April 5, 2024 09:25

Merge branch 'main' into use-pacta.data.validation

202d6ca

cjyetman marked this pull request as draft April 5, 2024 09:26

cjyetman added 2 commits April 11, 2024 20:09

Merge branch 'main' into use-pacta.data.validation

5d32809

Merge branch 'main' into use-pacta.data.validation

b3c9aec

cjyetman mentioned this pull request Apr 17, 2024

integrate {pacta.data.validation} into other PACTA pkgs and workflows RMI-PACTA/pacta.data.validation#38

Closed

Merge branch 'main' into use-pacta.data.validation

fc1f8b0

cjyetman added 2 commits April 18, 2024 09:14

remove financial_data validation

15160a3

remove abcd_flags_equity validation

c1afbe9

This was referenced Apr 18, 2024

allow technically invalid, but likely intentional ISINs with a warning RMI-PACTA/pacta.data.validation#69

Closed

implement validation for financial_data and abcd_flags_equity when ready #222

Open

cjyetman marked this pull request as ready for review April 18, 2024 07:46

cjyetman requested a review from jdhoffa April 18, 2024 07:46

jdhoffa approved these changes Apr 18, 2024

View reviewed changes

cjyetman merged commit a837c76 into main Apr 18, 2024
1 check passed

cjyetman deleted the use-pacta.data.validation branch April 18, 2024 09:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

implement `pacta.data.validation` #185

implement `pacta.data.validation` #185

cjyetman commented Mar 7, 2024 •

edited

Loading

jdhoffa commented Mar 7, 2024

jdhoffa commented Mar 7, 2024 •

edited

Loading

jdhoffa commented Mar 7, 2024 •

edited

Loading

jdhoffa commented Mar 7, 2024

cjyetman commented Apr 17, 2024 •

edited

Loading

implement pacta.data.validation #185

implement pacta.data.validation #185

Conversation

cjyetman commented Mar 7, 2024 • edited Loading

jdhoffa commented Mar 7, 2024

jdhoffa commented Mar 7, 2024 • edited Loading

jdhoffa commented Mar 7, 2024 • edited Loading

jdhoffa commented Mar 7, 2024

cjyetman commented Apr 17, 2024 • edited Loading

implement `pacta.data.validation` #185

implement `pacta.data.validation` #185

cjyetman commented Mar 7, 2024 •

edited

Loading

jdhoffa commented Mar 7, 2024 •

edited

Loading

jdhoffa commented Mar 7, 2024 •

edited

Loading

cjyetman commented Apr 17, 2024 •

edited

Loading