Activation Relaxation: A Local Dynamical Approximation to Backpropagation in the Brain
arXiv:2009.05359
Abstract
The backpropagation of error algorithm (backprop) has been instrumental in the recent success of deep learning. However, a key question remains as to whether backprop can be formulated in a manner suitable for implementation in neural circuitry. The primary challenge is to ensure that any candidate formulation uses only local information, rather than relying on global signals as in standard backprop. Recently several algorithms for approximating backprop using only local signals have been proposed. However, these algorithms typically impose other requirements which challenge biological plausibility: for example, requiring complex and precise connectivity schemes, or multiple sequential backwards phases with information being stored across phases. Here, we propose a novel algorithm, Activation Relaxation (AR), which is motivated by constructing the backpropagation gradient as the equilibrium point of a dynamical system. Our algorithm converges rapidly and robustly to the correct backpropagation gradients, requires only a single type of computational unit, utilises only a single parallel backwards relaxation phase, and can operate on arbitrary computation graphs. We illustrate these properties by training deep neural networks on visual classification tasks, and describe simplifications to the algorithm which remove further obstacles to neurobiological implementation (for example, the weight-transport problem, and the use of nonlinear derivatives), while preserving performance.
arXiv commentinitial upload; revised version (updated abstract, related work) 28-09-20; 05/10/20: revised for ICLR submission; 10/10/20: minor revisions
References in corpus (19)
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Language Models are Few-Shot Learners
- Scaling Laws for Neural Language Models
- Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition
- Learning Latent Dynamics for Planning from Pixels
- A Review of automatic differentiation and its efficient implementation
- Direct Feedback Alignment Provides Learning in Deep Neural Networks
- Jukebox: A Generative Model for Music
- Random feedback weights support learning in deep neural networks
- A Theory of Local Learning, the Learning Channel, and the Optimality of Backpropagation
- Deep Learning without Weight Transport
- A Theoretical Framework for Target Propagation
- Conducting Credit Assignment by Aligning Local Representations
- Generalization of Equilibrium Propagation to Vector Field Dynamics
- Early Inference in Energy-Based Models Approximates Back-Propagation
- Relaxing the Constraints on Predictive Coding Models
- Learning to Adapt by Minimizing Discrepancy
- Ghost Units Yield Biologically Plausible Backprop in Deep Neural Networks