256 citations · 436 across the 6 of their papers we have counts for
8 papers · 1 filter
Regularising for invariance to data augmentation improves supervised learning
Aleksander Botev, Matthias Bauer, Soham De
Data augmentation is used in machine learning to make the classifier invariant to label-preserving transformations. Usually this invariance is only encouraged implicitly by includi…
On the Origin of Implicit Regularization in Stochastic Gradient Descent
Samuel L. Smith, Benoit Dherin, David G. T. Barrett +1
For infinitesimal learning rates, stochastic gradient descent (SGD) follows the path of gradient flow on the full batch loss function. However moderately large learning rates can a…
Characterizing signal propagation to close the performance gap in unnormalized ResNets
Andrew Brock, Soham De, Samuel L. Smith
Batch Normalization is a key component in almost all state-of-the-art image classifiers, but it also introduces practical challenges: it breaks the independence between training ex…
On the Generalization Benefit of Noise in Stochastic Gradient Descent
Samuel L. Smith, Erich Elsen, Soham De
It has long been argued that minibatch stochastic gradient descent can generalize better than large batch gradient descent in deep neural networks. However recent papers have quest…
Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks
Soham De, Samuel L. Smith
Batch normalization dramatically increases the largest trainable depth of residual networks, and this benefit has been crucial to the empirical success of deep residual networks on…
The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent
Karthik A. Sankararaman, Soham De, Zheng Xu +2
This paper studies how neural network architecture affects the speed of training. We introduce a simple concept called gradient confusion to help formally analyze this. When gradie…