Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures
arXiv:2006.12878
Abstract
Despite being the workhorse of deep learning, the backpropagation algorithm is no panacea. It enforces sequential layer updates, thus preventing efficient parallelization of the training process. Furthermore, its biological plausibility is being challenged. Alternative schemes have been devised; yet, under the constraint of synaptic asymmetry, none have scaled to modern deep learning tasks and architectures. Here, we challenge this perspective, and study the applicability of Direct Feedback Alignment to neural view synthesis, recommender systems, geometric learning, and natural language processing. In contrast with previous studies limited to computer vision tasks, our findings show that it successfully trains a large range of state-of-the-art deep learning architectures, with performance close to fine-tuned backpropagation. At variance with common beliefs, our work supports that challenging tasks can be tackled in the absence of weight transport.
arXiv comment23 pages, 6 figures, 10 tables. For associated code, see https://github.com/lightonai/dfa-scales-to-modern-deep-learning. Poster at NeurIPS 2020
References in corpus (30)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Geometric deep learning: going beyond Euclidean data
- Language Models are Few-Shot Learners
- Learning deep representations by mutual information estimation and maximization
- Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
- Deep Convolutional Networks on Graph-Structured Data
- Fast Graph Representation Learning with PyTorch Geometric
- Variational Graph Auto-Encoders
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- DeepFM: A Factorization-Machine based Neural Network for CTR Prediction
- NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
- Pointer Sentinel Mixture Models
- Image Transformer
- Synthesizer: Rethinking Self-Attention in Transformer Models
- Direct Feedback Alignment Provides Learning in Deep Neural Networks
- Training Neural Networks with Local Error Signals
- How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation
- Jukebox: A Generative Model for Music
- Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures
- Putting An End to End-to-End: Gradient-Isolated Learning of Representations
- Deep Learning without Weight Transport
- How Important is Weight Symmetry in Backpropagation?
- Feedback alignment in deep convolutional networks
- Biologically-plausible learning algorithms can scale to large datasets
- Learning to solve the credit assignment problem
- Principled Training of Neural Networks with Direct Feedback Alignment
- Just Jump: Dynamic Neighborhood Aggregation in Graph Neural Networks
- Two Routes to Scalable Credit Assignment without Weight Symmetry
- Light-in-the-loop: using a photonics co-processor for scalable training of neural networks
- On the Morality of Artificial Intelligence