Feedback alignment in deep convolutional networks

Abstract

Ongoing studies have identified similarities between neural representations in biological networks and in deep artificial neural networks. This has led to renewed interest in developing analogies between the backpropagation learning algorithm used to train artificial networks and the synaptic plasticity rules operative in the brain. These efforts are challenged by biologically implausible features of backpropagation, one of which is a reliance on symmetric forward and backward synaptic weights. A number of methods have been proposed that do not rely on weight symmetry but, thus far, these have failed to scale to deep convolutional networks and complex data. We identify principal obstacles to the scalability of such algorithms and introduce several techniques to mitigate them. We demonstrate that a modification of the feedback alignment method that enforces a weaker form of weight symmetry, one that requires agreement of weight sign but not magnitude, can achieve performance competitive with backpropagation. Our results complement those of Bartunov et al. (2018) and Xiao et al. (2018b) and suggest that mechanisms that promote alignment of feedforward and feedback weights are critical for learning in deep networks.

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Striving for Simplicity: The All Convolutional Net
• Direct Feedback Alignment Provides Learning in Deep Neural Networks
• How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation
• Dendritic error backpropagation in deep cortical microcircuits
• Biologically-plausible learning algorithms can scale to large datasets
• Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks
• Learning in the Machine: Random Backpropagation and the Deep Learning Channel

Cited by in corpus (18)

• Biologically plausible deep learning -- but how far can we go with shallow networks?
• Deep Learning without Weight Transport
• A Theoretical Framework for Target Propagation
• Learning to solve the credit assignment problem
• Two Routes to Scalable Credit Assignment without Weight Symmetry
• Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks
• Spike-based causal inference for weight alignment
• Towards Biologically Plausible Convolutional Networks
• Biological credit assignment through dynamic inversion of feedforward networks
• Identifying Learning Rules From Neural Network Observables
• Differentially Private Deep Learning with Direct Feedback Alignment
• Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures
• Credit Assignment Through Broadcasting a Global Error Vector
• Benchmarking the Accuracy and Robustness of Feedback Alignment Algorithms
• Training DNNs in O(1) memory with MEM-DFA using Random Matrices
• Tourbillon: a Physically Plausible Neural Architecture
• How and When Random Feedback Works: A Case Study of Low-Rank Matrix Factorization
• Convergence Analysis and Implicit Regularization of Feedback Alignment for Deep Linear Networks