Principled Training of Neural Networks with Direct Feedback Alignment

Abstract

The backpropagation algorithm has long been the canonical training method for neural networks. Modern paradigms are implicitly optimized for it, and numerous guidelines exist to ensure its proper use. Recently, synthetic gradients methods -where the error gradient is only roughly approximated - have garnered interest. These methods not only better portray how biological brains are learning, but also open new computational possibilities, such as updating layers asynchronously. Even so, they have failed to scale past simple tasks like MNIST or CIFAR-10. This is in part due to a lack of standards, leading to ill-suited models and practices forbidding such methods from performing to the best of their abilities. In this work, we focus on direct feedback alignment and present a set of best practices justified by observations of the alignment angles. We characterize a bottleneck effect that prevents alignment in narrow layers, and hypothesize it may explain why feedback alignment methods have yet to scale to large convolutional networks.

arXiv comment10 pages, 4 figures, 4 tables, github repo at: https://github.com/lightonai/principled-dfa-training

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation
• Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures
• Decoupled Neural Interfaces using Synthetic Gradients
• Deep Learning without Weight Transport
• How Important is Weight Symmetry in Backpropagation?

Cited by in corpus (8)

• Bottom-up and top-down approaches for the design of neuromorphic processing systems: Tradeoffs and synergies between natural and artificial intelligence
• Learning without feedback: Fixed random learning signals allow for feedforward training of deep neural networks
• A Theoretical Framework for Target Propagation
• Adversarial Robustness by Design through Analog Computing and Synthetic Gradients
• Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures
• Credit Assignment Through Broadcasting a Global Error Vector
• Investigating the Scalability and Biological Plausibility of the Activation Relaxation Algorithm
• Convergence Analysis and Implicit Regularization of Feedback Alignment for Deep Linear Networks