Meta-Learning Bidirectional Update Rules
arXiv:2104.04657
Abstract
In this paper, we introduce a new type of generalized neural network where neurons and synapses maintain multiple states. We show that classical gradient-based backpropagation in neural networks can be seen as a special case of a two-state network where one state is used for activations and another for gradients, with update rules derived from the chain rule. In our generalized framework, networks have neither explicit notion of nor ever receive gradients. The synapses and neurons are updated using a bidirectional Hebb-style update rule parameterized by a shared low-dimensional "genome". We show that such genomes can be meta-learned from scratch, using either conventional optimization techniques, or evolutionary strategies, such as CMA-ES. Resulting update rules generalize to unseen tasks and train faster than gradient descent based optimizers for several standard computer vision and synthetic tasks.
arXiv commentICML 2021, 17 pages
References in corpus (28)
- Scikit-learn: Machine Learning in Python
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Deep Residual Learning for Image Recognition
- Instance Normalization: The Missing Ingredient for Fast Stylization
- EMNIST: an extension of MNIST to handwritten letters
- Towards Biologically Plausible Deep Learning
- Direct Feedback Alignment Provides Learning in Deep Neural Networks
- Training Neural Networks Without Gradients: A Scalable ADMM Approach
- AutoML-Zero: Evolving Machine Learning Algorithms From Scratch
- Born to Learn: the Inspiration, Progress, and Future of Evolved Plastic Artificial Neural Networks
- How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation
- Learned Optimizers that Scale and Generalize
- Putting An End to End-to-End: Gradient-Isolated Learning of Representations
- Deep Learning without Weight Transport
- Understanding and correcting pathologies in the training of learned optimizers
- Diffusion-based neuromodulation can eliminate catastrophic forgetting in simple neural networks
- Biologically-plausible learning algorithms can scale to large datasets
- Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity
- Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers
- Meta-Learning through Hebbian Plasticity in Random Networks
- LoCo: Local Contrastive Representation Learning
- Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
- Meta Learning Backpropagation And Improving It
- Notes on Chain Recurrence and Lyapunonv Functions
- Continual Learning with Deep Artificial Neurons
- MPLP: Learning a Message Passing Learning Protocol
- Reverse engineering learned optimizers reveals known and novel mechanisms
- Neuromodulated Learning in Deep Neural Networks