Knowledge-Adaptation Priors
arXiv:2106.08769
Abstract
Humans and animals have a natural ability to quickly adapt to their surroundings, but machine-learning models, when subjected to changes, often require a complete retraining from scratch. We present Knowledge-adaptation priors (K-priors) to reduce the cost of retraining by enabling quick and accurate adaptation for a wide-variety of tasks and models. This is made possible by a combination of weight and function-space priors to reconstruct the gradients of the past, which recovers and generalizes many existing, but seemingly-unrelated, adaptation strategies. Training with simple first-order gradient methods can often recover the exact retrained model to an arbitrary accuracy by choosing a sufficiently large memory of the past data. Empirical results show that adaptation with K-priors achieves performance similar to full retraining, but only requires training on a handful of past examples.
References in corpus (13)
- Distilling the Knowledge in a Neural Network
- Overcoming catastrophic forgetting in neural networks
- Continual Learning Through Synaptic Intelligence
- Challenges in Deploying Machine Learning: a Survey of Case Studies
- New insights and perspectives on the natural gradient method
- Overcoming Catastrophic Forgetting by Incremental Moment Matching
- Functional Regularisation for Continual Learning with Gaussian Processes
- Making AI Forget You: Data Deletion in Machine Learning
- Fast Randomized Kernel Methods With Statistical Guarantees
- Knowledge Adaptation: Teaching to Adapt
- Continual Learning in Practice
- Infinite-Horizon Gaussian Processes
- Machine Unlearning for Random Forests