Artificial Neural Variability for Deep Learning: On Overfitting, Noise Memorization, and Catastrophic Forgetting

Abstract

Deep learning is often criticized by two serious issues which rarely exist in natural nervous systems: overfitting and catastrophic forgetting. It can even memorize randomly labelled data, which has little knowledge behind the instance-label pairs. When a deep network continually learns over time by accommodating new tasks, it usually quickly overwrites the knowledge learned from previous tasks. Referred to as the {\it neural variability}, it is well-known in neuroscience that human brain reactions exhibit substantial variability even in response to the same stimulus. This mechanism balances accuracy and plasticity/flexibility in the motor learning of natural nervous systems. Thus it motivates us to design a similar mechanism named {\it artificial neural variability} (ANV), which helps artificial neural networks learn some advantages from ``natural'' neural networks. We rigorously prove that ANV plays as an implicit regularizer of the mutual information between the training data and the learned model. This result theoretically guarantees ANV a strictly improved generalizability, robustness to label noise, and robustness to catastrophic forgetting. We then devise a {\it neural variable risk minimization} (NVRM) framework and {\it neural variable optimizers} to achieve ANV for conventional network architectures in practice. The empirical studies demonstrate that NVRM can effectively relieve overfitting, label noise memorization, and catastrophic forgetting at negligible costs. \footnote{Code: \url{https://github.com/zeke-xie/artificial-neural-variability-for-deep-learning}.

arXiv commentAccepted by Neural Computation, MIT Press;20 pages; 13 figures; Key Words: Neural Variability, Neuroscience, Deep Learning, Label Noise, Catastrophic Forgetting

References in corpus (22)

• Adam: A Method for Stochastic Optimization
• A Survey on Deep Learning in Medical Image Analysis
• Decoupled Weight Decay Regularization
• Overcoming catastrophic forgetting in neural networks
• Weight Uncertainty in Neural Networks
• Continual Learning Through Synaptic Intelligence
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• A Closer Look at Memorization in Deep Networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Adding Gradient Noise Improves Learning for Very Deep Networks
• Three Factors Influencing Minima in SGD
• Information-theoretic analysis of generalization capability of learning algorithms
• Sharp Minima Can Generalize For Deep Nets
• The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects
• Where is the Information in a Deep Neural Network?
• Effect of Depth and Width on Local Minima in Deep Learning
• SmoothOut: Smoothing Out Sharp Minima to Improve Generalization in Deep Learning
• Robustness of Bayesian Neural Networks to Gradient-Based Attacks
• Towards Understanding the Importance of Noise in Training Neural Networks
• Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalization
• Improving Generalization by Controlling Label-Noise Information in Neural Network Weights

Cited by in corpus (2)

• A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima
• A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix