Contrastive Weight Regularization for Large Minibatch SGD

Abstract

The minibatch stochastic gradient descent method (SGD) is widely applied in deep learning due to its efficiency and scalability that enable training deep networks with a large volume of data. Particularly in the distributed setting, SGD is usually applied with large batch size. However, as opposed to small-batch SGD, neural network models trained with large-batch SGD can hardly generalize well, i.e., the validation accuracy is low. In this work, we introduce a novel regularization technique, namely distinctive regularization (DReg), which replicates a certain layer of the deep network and encourages the parameters of both layers to be diverse. The DReg technique introduces very little computation overhead. Moreover, we empirically show that optimizing the neural network with DReg using large-batch SGD achieves a significant boost in the convergence and improved generalization performance. We also demonstrate that DReg can boost the convergence of large-batch SGD with momentum. We believe that DReg can be used as a simple regularization trick to accelerate large-batch training in deep learning.

References in corpus (14)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Wide Residual Networks
• One weird trick for parallelizing convolutional neural networks
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Revisiting Small Batch Training for Deep Neural Networks
• Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects
• ImageNet Training in Minutes
• Large Margin Few-Shot Learning
• Stochastic Variance-Reduced Cubic Regularization for Nonconvex Optimization
• An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise
• Adaptive Loss Scaling for Mixed Precision Training