Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: compress all output json files into a archive.zip #27

Closed
jdhoffa opened this issue Nov 28, 2024 · 7 comments · Fixed by #36 or #45
Closed

feat: compress all output json files into a archive.zip #27

jdhoffa opened this issue Nov 28, 2024 · 7 comments · Fixed by #36 or #45
Assignees

Comments

@jdhoffa
Copy link
Member

jdhoffa commented Nov 28, 2024

In the dashboard, we wish to allow users to download the input datasets. This would facilitate that.

@cjyetman
Copy link
Member

cjyetman commented Dec 4, 2024

I think this was resolved by #82

@jdhoffa
Copy link
Member Author

jdhoffa commented Dec 4, 2024

It was not.
In RMI-PACTA/pacta-dashboard-svelte#82 (a different repo) a button was created that allows users to download a .zip archive.

This issue tracks the actual creation of that .zip archive (if it doesn't exist, the button won't do anything).

Acceptance criteria:

  • workflow.pacta.dashboard creates a .zip of all .json assets after a run is complete
  • (Nice-to-have) the .zip contains .csv files rather than .json files (probably more user friendly)

@cjyetman
Copy link
Member

cjyetman commented Dec 4, 2024

huh... currently on https://rmi-pacta.github.io/pacta-dashboard-svelte/ if I click the "Save Data" button it downloads an "archive.zip" which contains JSON files.

@jdhoffa
Copy link
Member Author

jdhoffa commented Dec 4, 2024

Indeed, because the archive.zip was manually created and merged into the sample-data directory that is used to spin up that dashboard.
It is out of sync with this repo, that's what this tracking issue is for 😊

@AlexAxthelm
Copy link
Contributor

Disabled as part of #35

@jdhoffa
Copy link
Member Author

jdhoffa commented Dec 9, 2024

I was able to locally reproduce the error mentioned in #35 :

workflow.pacta.dashboard-1  | Error in write_delim(x, file, delim = ",", na = na, append = append, col_names = col_names,  : 
workflow.pacta.dashboard-1  |   is.data.frame(x) is not TRUE
workflow.pacta.dashboard-1  | Calls: <Anonymous> ... zip_outputs -> %>% -> <Anonymous> -> write_delim -> stopifnot
workflow.pacta.dashboard-1  | Execution halted
workflow.pacta.dashboard-1 exited with code 1

Investigating.

@jdhoffa
Copy link
Member Author

jdhoffa commented Dec 9, 2024

The error occurs because the datasets data_company_bubbles.json and data_techexposure.json are empty.

These empty JSON files are then read into R as a list() object, and the list object cannot be written to a .csv file.

My recommendation here would be to add a check prior to calling jsonlite::write_json() every time in the script, and to not write an empty data frame in the first place, e.g. here:

audit_file %>%
prep_audit_table(
investor_name = investor_name,
portfolio_name = portfolio_name,
currency_exchange_value = currency_exchange_value
) %>%
translate_df_contents("data_included_table", dictionary, inplace = TRUE) %>%
translate_df_headers("data_included_table", language_select, header_dictionary) %>%
jsonlite::write_json(path = file.path(output_dir, "data_included_table.json"))

AlexAxthelm added a commit that referenced this issue Dec 10, 2024
re-enable zip output and csv conversion.

Note: if a JSON file is does not read as a data.frame (empty array,
`[]`), then this will not emit a corresponding csv file, but will add
the empty JSON to the archive (for later diagnostic purposes).

Closes: #27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants