Supermasks in Superposition
arXiv:2006.14769
Abstract
We present the Supermasks in Superposition (SupSup) model, capable of sequentially learning thousands of tasks without catastrophic forgetting. Our approach uses a randomly initialized, fixed base network and for each task finds a subnetwork (supermask) that achieves good performance. If task identity is given at test time, the correct subnetwork can be retrieved with minimal memory usage. If not provided, SupSup can infer the task using gradient-based optimization to find a linear superposition of learned supermasks which minimizes the output entropy. In practice we find that a single gradient step is often sufficient to identify the correct mask, even among 2500 tasks. We also showcase two promising extensions. First, SupSup models can be trained entirely without task identity information, as they may detect when they are uncertain about new data and allocate an additional supermask for the new training distribution. Finally the entire, growing set of supermasks can be stored in a constant-sized reservoir by implicitly storing them as attractors in a fixed-sized Hopfield network.
arXiv commentNeurIPS 2020 Camera Ready
References in corpus (22)
- Adam: A Method for Stochastic Optimization
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Overcoming catastrophic forgetting in neural networks
- On Calibration of Modern Neural Networks
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- Momentum Contrast for Unsupervised Visual Representation Learning
- Continual Learning Through Synaptic Intelligence
- An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
- Searching for Activation Functions
- Efficient Lifelong Learning with A-GEM
- Three scenarios for continual learning
- Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science
- Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization
- Reinforced Continual Learning
- Sparse Networks from Scratch: Faster Training without Losing Performance
- Lifelong Learning with Dynamically Expandable Networks
- Continual Learning via Neural Pruning
- Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One
- Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs
- Proving the Lottery Ticket Hypothesis: Pruning is All You Need
- Task Agnostic Continual Learning Using Online Variational Bayes
- Superposition of many models into one