A Theoretical Framework for Target Propagation
arXiv:2006.14331
Abstract
The success of deep learning, a brain-inspired form of AI, has sparked interest in understanding how the brain could similarly learn across multiple layers of neurons. However, the majority of biologically-plausible learning algorithms have not yet reached the performance of backpropagation (BP), nor are they built on strong theoretical foundations. Here, we analyze target propagation (TP), a popular but not yet fully understood alternative to BP, from the standpoint of mathematical optimization. Our theory shows that TP is closely related to Gauss-Newton optimization and thus substantially differs from BP. Furthermore, our analysis reveals a fundamental limitation of difference target propagation (DTP), a well-known variant of TP, in the realistic scenario of non-invertible neural networks. We provide a first solution to this problem through a novel reconstruction loss that improves feedback weight training, while simultaneously introducing architectural flexibility by allowing for direct feedback connections from the output to each hidden layer. Our theory is corroborated by experimental results that show significant improvements in performance and in the alignment of forward weight updates with loss gradients, compared to DTP.
arXiv comment13 pages and 4 figures in main manuscript; 41 pages and 8 figures in supplementary material
References in corpus (22)
- Adam: A Method for Stochastic Optimization
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Tune: A Research Platform for Distributed Model Selection and Training
- Optimizing Neural Networks with Kronecker-factored Approximate Curvature
- Towards Biologically Plausible Deep Learning
- Direct Feedback Alignment Provides Learning in Deep Neural Networks
- full-FORCE: A Target-Based Method for Training Recurrent Networks
- How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation
- Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures
- Deep Learning without Weight Transport
- How Important is Weight Symmetry in Backpropagation?
- Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks
- Feedback alignment in deep convolutional networks
- Why are deep nets reversible: A simple theory, with implications for training
- Biologically-plausible learning algorithms can scale to large datasets
- Practical Gauss-Newton Optimisation for Deep Learning
- Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems
- Learning to solve the credit assignment problem
- Principled Training of Neural Networks with Direct Feedback Alignment
- Two Routes to Scalable Credit Assignment without Weight Symmetry
- Biological credit assignment through dynamic inversion of feedforward networks
Cited by in corpus (9)
- Bottom-up and top-down approaches for the design of neuromorphic processing systems: Tradeoffs and synergies between natural and artificial intelligence
- Beyond spiking networks: the computational advantages of dendritic amplification and input segregation
- Neural Networks with Recurrent Generative Feedback
- Activation Relaxation: A Local Dynamical Approximation to Backpropagation in the Brain
- Biological credit assignment through dynamic inversion of feedforward networks
- Error-based or target-based? A unifying framework for learning in recurrent spiking networks
- KCNet: An Insect-Inspired Single-Hidden-Layer Neural Network with Randomized Binary Weights for Prediction and Classification Tasks
- Investigating the Scalability and Biological Plausibility of the Activation Relaxation Algorithm
- Self Normalizing Flows