Large-Scale Gradient-Free Deep Learning with Recursive Local Representation Alignment

Abstract

Training deep neural networks on large-scale datasets requires significant hardware resources whose costs (even on cloud platforms) put them out of reach of smaller organizations, groups, and individuals. Backpropagation, the workhorse for training these networks, is an inherently sequential process that is difficult to parallelize. Furthermore, it requires researchers to continually develop various tricks, such as specialized weight initializations and activation functions, in order to ensure a stable parameter optimization. Our goal is to seek an effective, neuro-biologically-plausible alternative to backprop that can be used to train deep networks. In this paper, we propose a gradient-free learning procedure, recursive local representation alignment, for training large-scale neural architectures. Experiments with residual networks on CIFAR-10 and the large benchmark, ImageNet, show that our algorithm generalizes as well as backprop while converging sooner due to weight updates that are parallelizable and computationally less demanding. This is empirical evidence that a backprop-free algorithm can scale up to larger datasets.

Further revised submission -- main description of rec-LRA revamped and architecture-agnostic pseudo-code moved to appendix with additional results/derivation updates

References in corpus (24)

• Adam: A Method for Stochastic Optimization
• Decoupled Weight Decay Regularization
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• On the difficulty of training Recurrent Neural Networks
• Neural Architecture Search: A Survey
• Deeply-Supervised Nets
• Self-training with Noisy Student improves ImageNet classification
• All you need is a good init
• Direct Feedback Alignment Provides Learning in Deep Neural Networks
• Unsupervised Learning by Competing Hidden Units
• Training Neural Networks Without Gradients: A Scalable ADMM Approach
• Training Neural Networks with Local Error Signals
• How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation
• Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures
• Decoupled Neural Interfaces using Synthetic Gradients
• Deep Learning without Weight Transport
• How Important is Weight Symmetry in Backpropagation?
• Distributed optimization of deeply nested systems
• Biologically-plausible learning algorithms can scale to large datasets
• Deep Predictive Coding Networks
• Greedy Layerwise Learning Can Scale to ImageNet
• Conducting Credit Assignment by Aligning Local Representations
• Learning to Adapt by Minimizing Discrepancy
• Hebbian-Descent

Cited by in corpus (2)

• Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networks
• Front Contribution instead of Back Propagation