212 citations · 215 across the 2 of their papers we have counts for
2 papers
cs.LG2020★ 3 cited
Understanding the Effects of Data Parallelism and Sparsity on Neural Network Training
Namhoon Lee, Thalaiyasingam Ajanthan, Philip H. S. Torr +1
We study two factors in neural network training: data parallelism and sparsity; here, data parallelism means processing training data in parallel using distributed systems (or equi…
cs.LG2018★ 212 cited
Don't Use Large Mini-Batches, Use Local SGD
Tao Lin, Sebastian U. Stich, Kumar Kshitij Patel +1
Mini-batch stochastic gradient methods (SGD) are state of the art for distributed training of deep neural networks. Drastic increases in the mini-batch sizes have lead to key effic…