Feature/cele 78

[dvcorreia]

Ingestion script

• connect the validator to the ingestion to pre-validate the dataset to ingest
• process and convert eventually 3d and json data
  segmentation extraction has been added in its own subcommand (celegans extract ...).
• push files into the bucket using the pattern designed for the file storage
• update the database with the new data ingested
• develop a flag to avoid to re-upload files that already presents in the bucket (fast upload)
  implemented but there's still some issues with the amount of files we are dealing with, which takes a lot of time to upload.
• develop a flag to upload everything and eventually overwrite what in the bucket (new data entirely, wipe what stored remotely)
  if the hash between the files is the same, nothing happens. if the hash is different and overwrite is true, the new file will overwrite the remote one.
• add a flag to eventually remove everything from the bucket before the upload

Extra tasks:

• upload a file to the bucket with the tiles metadata
  we assume that all the tiles in a slice have the same size, otherwise it would take a lot of time to read the tile size from all tiles
  we also have to understand what we want in the metadata.
• ingestion format documentation
• check that the specified dataset id in the CLI exists in datasets.json
  the CLI allows the user to partial upload a dataset, so a combination of the data, EM tiles and segmentations.
  currently this check is only evaluated if the data flag is set, otherwise we do not have access to datasets.json
• update README with usage documentation

[aranega]

All looks good to me :) I'm being picky on the code, but they are good examples about how some python code can be rewritten to be more pythonic