Superposition of many models into one
arXiv:1902.05522
Abstract
We present a method for storing multiple models within a single set of parameters. Models can coexist in superposition and still be retrieved individually. In experiments with neural networks, we show that a surprisingly large number of models can be effectively stored within a single parameter instance. Furthermore, each of these models can undergo thousands of training steps without significantly interfering with other models within the superposition. This approach may be viewed as the online complement of compression: rather than reducing the size of a network after training, we make use of the unrealized capacity of a network during training.
References in corpus (8)
- Overcoming catastrophic forgetting in neural networks
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Playing Atari with Deep Reinforcement Learning
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- Understanding deep learning requires rethinking generalization
- An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
- Overcoming catastrophic forgetting with hard attention to the task
- Measuring the Intrinsic Dimension of Objective Landscapes
Cited by in corpus (14)
- A comparison of Vector Symbolic Architectures
- Once-for-All: Train One Network and Specialize it for Efficient Deployment
- Vector Symbolic Architectures as a Computing Framework for Emerging Hardware
- How fine can fine-tuning be? Learning efficient language models
- Beneficial Perturbation Network for designing general adaptive artificial intelligence systems
- Deep Ensembles for Low-Data Transfer Learning
- Supermasks in Superposition
- Orthogonal Convolutional Neural Networks
- Gaussian Gated Linear Networks
- Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inference
- On Effects of Compression with Hyperdimensional Computing in Distributed Randomized Neural Networks
- Hyperdimensional Computing for Efficient Distributed Classification with Randomized Neural Networks
- Transferable Graph Optimizers for ML Compilers
- Bayesian Sparsification Methods for Deep Complex-valued Networks