Parallelize distance calculation

[gordonkoehn]

The runtime of distance calculation may exceed the actual MCMC runtime.

For MP3 distance with 30 mutations in the trees, the runtime of calculating the distances is more than double that of the actual MCMC chain.

This calls for making the distance calculation parallelizable.

I.e. compute the distances in chunks using multiprocessing in Python.

In particular, the function to be parallelized is:

yg.analyze.analyze_mcmc_run(mcmc_data, metric, base_tree)

Along with this, the number of threads should be adjusted in the snakemake rules.

[gordonkoehn]

@pawel-czyz FYI - distance calculation takes quite a lot of time in our experiments I just realized.

[pawel-czyz]

This is a good point! (But I'm not sure if it should get a higher priority than SMC).

In terms of implementation, let's discuss, but to write down some ideas before I forget:

1. Indeed, Snakemake can be a good idea here.
2. Alternatively, joblib provides a very nice API allowing to wrap a function into a parallelized one. (Simpler than using multiprocessing.Pool.)

[gordonkoehn]

Yes, was thinking of the second one before. Used it in my last project a lot.

Absolutely, priority is low.