3 papers
cs.DC2020
Towards Scalable Distributed Training of Deep Learning on Public Cloud Clusters
Shaohuai Shi, Xianhao Zhou, Shutao Song +21
Distributed training techniques have been widely deployed in large-scale deep neural networks (DNNs) training on dense-GPU clusters. However, on public cloud clusters, due to the m…
cs.LG2019
Distributed Equivalent Substitution Training for Large-Scale Recommender Systems
Haidong Rong, Yangzihao Wang, Feihu Zhou +8
We present Distributed Equivalent Substitution (DES) training, a novel distributed training framework for large-scale recommender systems with dynamic sparse features. DES introduc…
cs.LG2018
Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
Xianyan Jia, Shutao Song, Wei He +11
Synchronized stochastic gradient descent (SGD) optimizers with data parallelism are widely used in training large-scale deep neural networks. Although using larger mini-batch sizes…