refactoring

[vuillaut]

Hi
Linked to discussion #113 .

To support different pipelines, we probably need a more generic and consistent approach.
At the end, the steps are always the same... (r0 to dl1, train, apply to test dataset, produce irfs...)

E.g. I would see a very generic method

lstmcpipe_core --pipe_script --input --output --slurm_options --pipe_args

Input and output could be directories or mapped file lists (TBD).

slurm_options:
- n_jobs_max
- dependencies
- hardware requirements
... 

From there, one could make e.g.
lstmcpipe_split_train_test, lstmcpipe_r0_to_dl1, lstmcpipe_dl1_to_dl2, lstmcpipe_train_from_dl1, lstmcpipe_dl2_to_irfs...

[LukasNickel]

If we construct this the right way, we might also be able to process real data with just some extra steps for pedestals, calibration data ...

[vuillaut]

Moving forward with this, the idea is first to separate the internal logic (steps order and dependencies, handling jobs, slurm config...) from the paths handling and stages options.

The idea is to be much more modular regarding paths, which are now relatively hard-coded and tight to the current MC productions (in config we provide things like prod_type, obs_date, zenith, pointing....)

lstmcpipe new config file would look like:

workflow_kind: lstchain

# prod_id ex: local_no_n_islands. Default; v00 (if key left empty or None)
prod_id: prodidv00

source_environment:
  source_file: /fefs/aswg/software/conda/etc/profile.d/conda.sh
  conda_env: lstchain-v0.9.3

slurm_config:
  user_account: aswg

stages_to_run:
  - r0_to_dl1
  - train_test_split

stages:
  r0_to_dl1:
    - input: /fefs/aswg/data/mc/DL0/20200629_prod5_trans_80/gamma-diffuse/zenith_20deg/south_pointing
    output: /fefs/aswg/data/mc/DL1/20200629_prod5_trans_80/gamma-diffuse/zenith_20deg/south_pointing/prodidv00
    - input: /fefs/aswg/data/mc/DL0/20200629_prod5_trans_80/proton/zenith_20deg/south_pointing
    output: /fefs/aswg/data/mc/DL1/20200629_prod5_trans_80/proton/zenith_20deg/south_pointing/prodidv00

  train_test_split:
    - input: /fefs/aswg/data/mc/DL1/20200629_prod5_trans_80/gamma-diffuse/zenith_20deg/south_pointing/prodidv00
    output:
      train: /fefs/aswg/data/mc/DL1/20200629_prod5_trans_80/gamma-diffuse/zenith_20deg/south_pointing/prodidv00/train
      test: /fefs/aswg/data/mc/DL1/20200629_prod5_trans_80/gamma-diffuse/zenith_20deg/south_pointing/prodidv00/test
    - input: /fefs/aswg/data/mc/DL1/20200629_prod5_trans_80/proton/zenith_20deg/south_pointing/prodidv00
    output:
      train: /fefs/aswg/data/mc/DL1/20200629_prod5_trans_80/proton/zenith_20deg/south_pointing/prodidv00/train
      test: /fefs/aswg/data/mc/DL1/20200629_prod5_trans_80/proton/zenith_20deg/south_pointing/prodidv00/test

This is being implemented in https://github.com/cta-observatory/lstmcpipe/tree/refactor

When this is done and in a second, we can more easily move to a workflow management system, since the stages logic has been decoupled from the core of lstmcpipe.