35 citations · 54 across the 2 of their papers we have counts for
3 papers
cs.DC2020★ 19 cited
Automatic Cross-Replica Sharding of Weight Update in Data-Parallel Training
Yuanzhong Xu, HyoukJoong Lee, Dehao Chen +3
In data-parallel synchronous training of deep neural networks, different devices (replicas) run the same program with different partitions of the training batch, but weight update…
cs.LG2019★ 35 cited
Scale MLPerf-0.6 models on Google TPU-v3 Pods
Sameer Kumar, Victor Bitorff, Dehao Chen +9
The recent submission of Google TPU-v3 Pods to the industry wide MLPerf v0.6 training benchmark demonstrates the scalability of a suite of industry relevant ML models. MLPerf defin…
cs.LG2018
Mesh-TensorFlow: Deep Learning for Supercomputers
Noam Shazeer, Youlong Cheng, Niki Parmar +9
Batch-splitting (data-parallelism) is the dominant distributed Deep Neural Network (DNN) training strategy, due to its universal applicability and its amenability to Single-Program…