Distributed Pytorch implementation

[tlin-taolin]

Here I've provided my simplified Pytorch code for distributed training. The MLBench was building on top of it and has quickly evolved. The MLBench now is a bit hard for the beginner to start with; maybe this code also has this issue but will continue to simplify it.

So in the next few days, I will try to introduce some of the new features/code design choices (e.g., different sparsification/quantization schemes) from MLBench, and make this code to as simple as possible.

Also, I still need to figure out how can I merge this code with Thijs's single worker code.

[martinjaggi]

@iamtao , can you simplify the code structure it a bit in view of thijs' single machine code? https://github.com/epfml/cifar/blob/master/train.py
or even better if it would look a bit like this smaller one here?
https://github.com/epfml/autoTrain/blob/master/src/train_adam.py

[tlin-taolin]

@iamtao , can you simplify the code structure it a bit in view of thijs' single machine code? https://github.com/epfml/cifar/blob/master/train.py
or even better if it would look a bit like this smaller one here?
https://github.com/epfml/autoTrain/blob/master/src/train_adam.py

Yes. I will try to simplify the current code and make it more close to Thijs's scheme.