Training Neural Networks with Local Error Signals

Abstract

Supervised training of neural networks for classification is typically performed with a global loss function. The loss function provides a gradient for the output layer, and this gradient is back-propagated to hidden layers to dictate an update direction for the weights. An alternative approach is to train the network with layer-wise loss functions. In this paper we demonstrate, for the first time, that layer-wise training can approach the state-of-the-art on a variety of image datasets. We use single-layer sub-networks and two different supervised loss functions to generate local error signals for the hidden layers, and we show that the combination of these losses help with optimization in the context of local learning. Using local errors could be a step towards more biologically plausible deep learning because the global error does not have to be transported back to hidden layers. A completely backprop free variant outperforms previously reported results among methods aiming for higher biological plausibility. Code is available https://github.com/anokland/local-loss

arXiv commentAccepted to ICML 2019

Cited by in corpus (42)

• SpinalNet: Deep Neural Network with Gradual Input
• Layer-wise training convolutional neural networks with smaller filters for human activity recognition using wearable sensors
• Why should we add early exits to neural networks?
• Bottom-up and top-down approaches for the design of neuromorphic processing systems: Tradeoffs and synergies between natural and artificial intelligence
• Putting An End to End-to-End: Gradient-Isolated Learning of Representations
• Learning without feedback: Fixed random learning signals allow for feedforward training of deep neural networks
• Drawing Inspiration from Biological Dendrites to Empower Artificial Neural Networks
• Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective
• Ensemble learning in CNN augmented with fully connected subnetworks
• Contrastive Self-supervised Neural Architecture Search
• Revisiting Locally Supervised Learning: an Alternative to End-to-end Training
• Self-Supervised Learning with Kernel Dependence Maximization
• Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks
• Local plasticity rules can learn deep representations using self-supervised contrastive predictions
• Predictive Coding Can Do Exact Backpropagation on Convolutional and Recurrent Neural Networks
• Fully Decoupled Neural Network Learning Using Delayed Gradients
• Hardware Beyond Backpropagation: a Photonic Co-Processor for Direct Feedback Alignment
• A General Framework for Interpretable Neural Learning based on Local Information-Theoretic Goal Functions
• Towards Biologically Plausible Convolutional Networks
• Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networks
• VINNAS: Variational Inference-based Neural Network Architecture Search
• Convex Regularization Behind Neural Reconstruction
• Faster Convergence in Deep-Predictive-Coding Networks to Learn Deeper Representations
• Gaussian Gated Linear Networks
• Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures
• Large-Scale Gradient-Free Deep Learning with Recursive Local Representation Alignment
• Credit Assignment Through Broadcasting a Global Error Vector
• Learning to Time-Decode in Spiking Neural Networks Through the Information Bottleneck
• Why Layer-Wise Learning is Hard to Scale-up and a Possible Solution via Accelerated Downsampling
• Learning in Deep Neural Networks Using a Biologically Inspired Optimizer
• An Analytic Layer-wise Deep Learning Framework with Applications to Robotics
• Signature-Graph Networks
• Questions to Guide the Future of Artificial Intelligence Research
• Train your classifier first: Cascade Neural Networks Training from upper layers to lower layers
• On-Chip Error-triggered Learning of Multi-layer Memristive Spiking Neural Networks
• Accumulated Decoupled Learning: Mitigating Gradient Staleness in Inter-Layer Model Parallelization
• Intra-Model Collaborative Learning of Neural Networks
• XConv: Low-memory stochastic backpropagation for convolutional layers
• The staircase property: How hierarchical structure can guide deep learning
• Associated Learning: Decomposing End-to-end Backpropagation based on Auto-encoders and Target Propagation
• Neighbourhood Distillation: On the benefits of non end-to-end distillation
• Front Contribution instead of Back Propagation