Direct Feedback Alignment Provides Learning in Deep Neural Networks

Abstract

Artificial neural networks are most commonly trained with the back-propagation algorithm, where the gradient for learning is provided by back-propagating the error, layer by layer, from the output layer to the hidden layers. A recently discovered method called feedback-alignment shows that the weights used for propagating the error backward don't have to be symmetric with the weights used for propagation the activation forward. In fact, random feedback weights work evenly well, because the network learns how to make the feedback useful. In this work, the feedback alignment principle is used for training hidden layers more independently from the rest of the network, and from a zero initial condition. The error is propagated through fixed random feedback connections directly from the output layer to each hidden layer. This simple method is able to achieve zero training error even in convolutional networks and very deep networks, completely without error back-propagation. The method is a step towards biologically plausible machine learning because the error signal is almost local, and no symmetric or reciprocal weights are required. Experiments show that the test performance on MNIST and CIFAR is almost as good as those obtained with back-propagation for fully connected networks. If combined with dropout, the method achieves 1.45% error on the permutation invariant MNIST task.

Accepted for publication at NIPS 2016. [v2] Corrected convolutional results for feedback-alignment. [v3,v4,v5] Corrected theorem and proof

References in corpus (4)

• Explaining and Harnessing Adversarial Examples
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Towards Biologically Plausible Deep Learning
• Random feedback weights support learning in deep neural networks

Cited by in corpus (50)

• Experimentally realized in situ backpropagation for deep learning in nanophotonic neural networks
• Neuromorphic Deep Learning Machines
• Unsupervised Learning by Competing Hidden Units
• Biologically plausible deep learning -- but how far can we go with shallow networks?
• A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Services
• Bottom-up and top-down approaches for the design of neuromorphic processing systems: Tradeoffs and synergies between natural and artificial intelligence
• Silicon Photonic Architecture for Training Deep Neural Networks with Direct Feedback Alignment
• Learning without feedback: Fixed random learning signals allow for feedforward training of deep neural networks
• Decoupled Neural Interfaces using Synthetic Gradients
• Deep Neural Generative Model of Functional MRI Images for Psychiatric Disorder Diagnosis
• Training Neural Networks Using Features Replay
• Feedback alignment in deep convolutional networks
• A Theoretical Framework for Target Propagation
• Understanding Synthetic Gradients and Decoupled Neural Interfaces
• Learning to solve the credit assignment problem
• Metalearned Neural Memory
• SoftHebb: Bayesian Inference in Unsupervised Hebbian Soft Winner-Take-All Networks
• Two Routes to Scalable Credit Assignment without Weight Symmetry
• Conducting Credit Assignment by Aligning Local Representations
• LoCo: Local Contrastive Representation Learning
• ETLP: Event-based Three-factor Local Plasticity for online learning with neuromorphic hardware
• Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks
• Relaxing the Constraints on Predictive Coding Models
• SPIDE: A Purely Spike-based Method for Training Feedback Spiking Neural Networks
• Memorized Sparse Backpropagation
• Fully Decoupled Neural Network Learning Using Delayed Gradients
• Activation Relaxation: A Local Dynamical Approximation to Backpropagation in the Brain
• Learning to Adapt by Minimizing Discrepancy
• Gradient-Free Training of Recurrent Neural Networks using Random Perturbations
• Adversarial Robustness by Design through Analog Computing and Synthetic Gradients
• Biological credit assignment through dynamic inversion of feedforward networks
• Identifying Learning Rules From Neural Network Observables
• An Accelerated Analog Neuromorphic Hardware System Emulating NMDA- and Calcium-Based Non-Linear Dendrites
• DANTE: Deep AlterNations for Training nEural networks
• Adaptive Bidirectional Backpropagation: Towards Biologically Plausible Error Signal Transmission in Neural Networks
• Large-Scale Gradient-Free Deep Learning with Recursive Local Representation Alignment
• Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures
• Backprop-Q: Generalized Backpropagation for Stochastic Computation Graphs
• Meta-Learning Bidirectional Update Rules
• Benchmarking the Accuracy and Robustness of Feedback Alignment Algorithms
• Scaling of hardware-compatible perturbative training algorithms
• Accelerated CNN Training Through Gradient Approximation
• Learning in the Machine: the Symmetries of the Deep Learning Channel
• Contrastive Similarity Matching for Supervised Learning
• Investigating the Scalability and Biological Plausibility of the Activation Relaxation Algorithm
• Tourbillon: a Physically Plausible Neural Architecture
• CATERPILLAR: Coarse Grain Reconfigurable Architecture for Accelerating the Training of Deep Neural Networks
• Convergence Analysis and Implicit Regularization of Feedback Alignment for Deep Linear Networks
• Convergence guarantees for forward gradient descent in the linear regression model
• Associated Learning: Decomposing End-to-end Backpropagation based on Auto-encoders and Target Propagation