Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalization

Abstract

It is well-known that stochastic gradient noise (SGN) acts as implicit regularization for deep learning and is essentially important for both optimization and generalization of deep networks. Some works attempted to artificially simulate SGN by injecting random noise to improve deep learning. However, it turned out that the injected simple random noise cannot work as well as SGN, which is anisotropic and parameter-dependent. For simulating SGN at low computational costs and without changing the learning rate or batch size, we propose the Positive-Negative Momentum (PNM) approach that is a powerful alternative to conventional Momentum in classic optimizers. The introduced PNM method maintains two approximate independent momentum terms. Then, we can control the magnitude of SGN explicitly by adjusting the momentum difference. We theoretically prove the convergence guarantee and the generalization advantage of PNM over Stochastic Gradient Descent (SGD). By incorporating PNM into the two conventional optimizers, SGD with Momentum and Adam, our extensive experiments empirically verified the significant advantage of the PNM-based variants over the corresponding conventional Momentum-based optimizers.

arXiv commentICML 2021; 20 pages; 13 figures; We fixed some typos in the updated version

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Decoupled Weight Decay Regularization
• Recurrent Neural Network Regularization
• On the Convergence of Adam and Beyond
• Don't Decay the Learning Rate, Increase the Batch Size
• Revisiting Small Batch Training for Deep Neural Networks
• Adding Gradient Noise Improves Learning for Very Deep Networks
• Three Factors Influencing Minima in SGD
• Adaptive Gradient Methods with Dynamic Bound of Learning Rate
• The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects
• Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
• Escaping Saddles with Stochastic Gradients
• On the Noisy Gradient Descent that Generalizes as SGD
• A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
• Escaping Saddle Points with Adaptive Gradient Methods
• Towards Understanding the Importance of Noise in Training Neural Networks

Cited by in corpus (2)

• Artificial Neural Variability for Deep Learning: On Overfitting, Noise Memorization, and Catastrophic Forgetting
• Enhance Diffusion to Improve Robust Generalization