Skip to content
This repository has been archived by the owner on May 18, 2023. It is now read-only.

Dataset cloud function preprocessing should handle improper datasets. #70

Open
harrykeightley opened this issue Oct 7, 2022 · 0 comments
Labels
bug Something isn't working good first issue Good for newcomers

Comments

@harrykeightley
Copy link
Collaborator

harrykeightley commented Oct 7, 2022

Currently the preprocessing workflow blindly attempts to create processing jobs from the files supplied in a dataset.
This would fail if the user selected improper files, not enough files etc.

Instead, after forming a dataset object from the incoming event, the process_dataset function should check dataset validity before batching and pushing jobs to the pubsub queue.

If the dataset is found to be invalid, we should indicate so on the dataset model in firestore (setting status to error or something).

@harrykeightley harrykeightley added the bug Something isn't working label Oct 7, 2022
@harrykeightley harrykeightley added the good first issue Good for newcomers label Oct 14, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

1 participant