Sync-Switch: Hybrid Parameter Synchronization for Distributed Deep Learning

Shijian Li, Oren Mangoubi, Lijie Xu +1

Stochastic Gradient Descent (SGD) has become the de facto way to train deep neural networks in distributed clusters. A critical factor in determining the training throughput and mo…