Week 5: Data-parallel training Lecture: slides, video Seminar: link, video Homework: see the homework folder Further reading Python multiprocessing docs (pay attention to fork vs spawn!) PyTorch Distributed tutorial Collective communication protocols in NCCL There's a ton of links on the slides, please check the PDF.