Overcoming Catastrophic Forgetting by Generative Regularization
arXiv:1912.01238
Abstract
In this paper, we propose a new method to overcome catastrophic forgetting by adding generative regularization to Bayesian inference framework. Bayesian method provides a general framework for continual learning. We could further construct a generative regularization term for all given classification models by leveraging energy-based models and Langevin-dynamic sampling to enrich the features learned in each task. By combining discriminative and generative loss together, we empirically show that the proposed method outperforms state-of-the-art methods on a variety of tasks, avoiding catastrophic forgetting in continual learning. In particular, the proposed method outperforms baseline methods over 15% on the Fashion-MNIST dataset and 10% on the CUB dataset
References in corpus (13)
- Overcoming catastrophic forgetting in neural networks
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Axiomatic Attribution for Deep Networks
- Wide Residual Networks
- Continual Learning Through Synaptic Intelligence
- PathNet: Evolution Channels Gradient Descent in Super Neural Networks
- Implicit Generation and Generalization in Energy-Based Models
- Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One
- Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting
- A Unifying Bayesian View of Continual Learning
- Improving and Understanding Variational Continual Learning
- SupportNet: solving catastrophic forgetting in class incremental learning with support data
- Exponential Family Estimation via Adversarial Dynamics Embedding