DropCompute: simple and more robust distributed synchronous training via compute variance reduction

Niv Giladi, Shahar Gottlieb, Moran Shkolnik +5

Background: Distributed training is essential for large scale training of deep neural networks (DNNs). The dominant methods for large scale DNN training are synchronous (e.g. All-R…