Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor Collection with HTTP GET Requests. Closes #272 #271

Closed
wants to merge 9 commits into from

Conversation

umesh-timalsina
Copy link
Contributor

@umesh-timalsina umesh-timalsina commented Sep 28, 2023

I have refactored the data collection mechanism in chimerapy-engine with the following changes:

  • The manager requests collection using HTTP post, and waits for the FSM to change to "SAVED"
  • A collect Event is emitted through the post request.
  • The record service teardown, then emits an ArtifactEvent with the necessary metadata needed to serve the file with HTTP GET from the worker's server.
  • The manager, then requests this information, and a series of aiohttp file download utilities are fired to download/copy move files

Todos

  • Test Long Running Pipelines
  • Support Images as zipped directory
  • Hash Checksums?
  • CleanUp old functions

@umesh-timalsina umesh-timalsina added task Chunk of work that is a small feature or part of a bigger feature do not merge wip labels Sep 28, 2023
@umesh-timalsina umesh-timalsina changed the title Refactor Collection with HTTP GET Requests Refactor Collection with HTTP GET Requests. Closes #272 Sep 28, 2023
@umesh-timalsina umesh-timalsina removed task Chunk of work that is a small feature or part of a bigger feature do not merge labels Oct 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Development

Successfully merging this pull request may close these issues.

1 participant