Preprocess: split multiple process pipes pre/post demod

[msilvafe]

We need the ability to run multiple preprocess pipelines that each have unique config files and databases and are dependent on one another. The main use case is splitting the SAT preprocessing into pre- and post-demodulation pipelines. The idea being that all cuts, flags, and filters pre-demod will most likely live 100% inside of the flp/site-pipeline framework however there are multiple ideas for filtering and mapmaking after demodulation which would require different post-demod pipelines (or not use the preprocessing module at all post-demod). The tricky points that will need some development/infrastructure work I can see with this are:

1. The driving scripts are either load_preprocess_tod which pass a proc_aman to pipe.run or preprocess_tod which passes no proc_aman so there's actually no way to have the output axis manager of one load_preprocess_tod or preprocess_tod to be passed to a new call of pipe.run without any proc_aman we need a new driving script that takes in two config files with unique databases and writes to the second database (dependent on the processing in the first).
2. How to keep track of the multiple pipeline dependencies i.e. post-demod pipelines a, b, c are dependent on pre-demod pipeline d
3. How to keep flags that are computed in post-demod steps but when only loading in processing up through demod.

[iparask]

2. How to keep track of the multiple pipeline dependencies i.e. post-demod pipelines a, b, c are dependent on pre-demod pipeline d

You can either use a workflow management system that provides the capabilities to define dependencies, or use mpi4py futures to create sid dependencies.

[msilvafe]

2. How to keep track of the multiple pipeline dependencies i.e. post-demod pipelines a, b, c are dependent on pre-demod pipeline d

You can either use a workflow management system that provides the capabilities to define dependencies, or use mpi4py futures to create sid dependencies.

Thanks @iparask! I don't think I need mpi so I'll skip mpi4py for now but can you give me an example of a suitable workflow management system?