Convergence Analysis and Implicit Regularization of Feedback Alignment for Deep Linear Networks

Abstract

We theoretically analyze the Feedback Alignment (FA) algorithm, an efficient alternative to backpropagation for training neural networks. We provide convergence guarantees with rates for deep linear networks for both continuous and discrete dynamics. Additionally, we study incremental learning phenomena for shallow linear networks. Interestingly, certain specific initializations imply that negligible components are learned before the principal ones, thus potentially negatively affecting the effectiveness of such a learning algorithm; a phenomenon we classify as implicit anti-regularization. We also provide initialization schemes where the components of the problem are approximately learned by decreasing order of importance, thus providing a form of implicit regularization.

arXiv comment10 pages (Main) + 19 pages (Appendix), 6 figures

References in corpus (8)

• Direct Feedback Alignment Provides Learning in Deep Neural Networks
• Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures
• Learning without feedback: Fixed random learning signals allow for feedforward training of deep neural networks
• Feedback alignment in deep convolutional networks
• Principled Training of Neural Networks with Direct Feedback Alignment
• Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks
• The Implicit Bias of Depth: How Incremental Learning Drives Generalization
• Training DNNs in O(1) memory with MEM-DFA using Random Matrices