Improving and Understanding Variational Continual Learning
arXiv:1905.02099
Abstract
In the continual learning setting, tasks are encountered sequentially. The goal is to learn whilst i) avoiding catastrophic forgetting, ii) efficiently using model capacity, and iii) employing forward and backward transfer learning. In this paper, we explore how the Variational Continual Learning (VCL) framework achieves these desiderata on two benchmarks in continual learning: split MNIST and permuted MNIST. We first report significantly improved results on what was already a competitive approach. The improvements are achieved by establishing a new best practice approach to mean-field variational Bayesian neural networks. We then look at the solutions in detail. This allows us to obtain an understanding of why VCL performs as it does, and we compare the solution to what an `ideal' continual learning solution might be.
References in corpus (10)
- Overcoming catastrophic forgetting in neural networks
- Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
- Continual Learning Through Synaptic Intelligence
- An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
- Bayesian Compression for Deep Learning
- Variational Continual Learning
- Generative replay with feedback connections as a general strategy for continual learning
- Deep Gaussian Processes for Regression using Approximate Expectation Propagation
- Continual Learning via Neural Pruning
- Overpruning in Variational Bayesian Neural Networks
Cited by in corpus (14)
- A continual learning survey: Defying forgetting in classification tasks
- Practical Deep Learning with Bayesian Principles
- LEEP: A New Measure to Evaluate Transferability of Learned Representations
- Continual Deep Learning by Functional Regularisation of Memorable Past
- On the Expressiveness of Approximate Inference in Bayesian Neural Networks
- Toward Understanding Catastrophic Forgetting in Continual Learning
- Neural Network Retraining for Model Serving
- Unifying Regularisation Methods for Continual Learning
- Posterior Meta-Replay for Continual Learning
- Overcoming Catastrophic Forgetting by Generative Regularization
- Frosting Weights for Better Continual Training
- Continual Learning from the Perspective of Compression
- Variational Auto-Regressive Gaussian Processes for Continual Learning
- TyXe: Pyro-based Bayesian neural nets for Pytorch