The Effect of Network Width on the Performance of Large-batch Training

Abstract

Distributed implementations of mini-batch stochastic gradient descent (SGD) suffer from communication overheads, attributed to the high frequency of gradient updates inherent in small-batch training. Training with large batches can reduce these overheads; however, large batches can affect the convergence properties and generalization performance of SGD. In this work, we take a first step towards analyzing how the structure (width and depth) of a neural network affects the performance of large-batch training. We present new theoretical results which suggest that--for a fixed number of parameters--wider networks are more amenable to fast large-batch training compared to deeper ones. We provide extensive experiments on residual and fully-connected neural networks which suggest that wider networks can be trained using larger batches without incurring a convergence slow-down, unlike their deeper variants.

Cited by in corpus (7)

• Measuring the Effects of Data Parallelism on Neural Network Training
• The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent
• Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model
• MicroRec: Efficient Recommendation Inference by Hardware and Data Structure Solutions
• SparCML: High-Performance Sparse Communication for Machine Learning
• Principal Component Networks: Parameter Reduction Early in Training
• Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updates