Learning without feedback: Fixed random learning signals allow for feedforward training of deep neural networks

Abstract

While the backpropagation of error algorithm enables deep neural network training, it implies (i) bidirectional synaptic weight transport and (ii) update locking until the forward and backward passes are completed. Not only do these constraints preclude biological plausibility, but they also hinder the development of low-cost adaptive smart sensors at the edge, as they severely constrain memory accesses and entail buffering overhead. In this work, we show that the one-hot-encoded labels provided in supervised classification problems, denoted as targets, can be viewed as a proxy for the error sign. Therefore, their fixed random projections enable a layerwise feedforward training of the hidden layers, thus solving the weight transport and update locking problems while relaxing the computational and memory requirements. Based on these observations, we propose the direct random target projection (DRTP) algorithm and demonstrate that it provides a tradeoff between accuracy and computational cost that is suitable for adaptive edge computing devices.

arXiv commentThis document is the paper as accepted for publication in the Frontiers in Neuroscience journal, the fully-edited paper is available at https://www.frontiersin.org/articles/10.3389/fnins.2021.629892

References in corpus (15)

• Scikit-learn: Machine Learning in Python
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• A 0.086-mm${}^2$ 12.7-pJ/SOP 64k-Synapse 256-Neuron Online-Learning Digital Spiking Neuromorphic Processor in 28nm CMOS
• Synaptic Plasticity Dynamics for Deep Continuous Local Learning (DECOLLE)
• Neuromorphic Deep Learning Machines
• Direct Feedback Alignment Provides Learning in Deep Neural Networks
• MorphIC: A 65-nm 738k-Synapse/mm${}^2$ Quad-Core Binary-Weight Digital Neuromorphic Processor with Stochastic Spike-Driven Online Learning
• Training Neural Networks with Local Error Signals
• Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures
• How Important is Weight Symmetry in Backpropagation?
• Understanding Synthetic Gradients and Decoupled Neural Interfaces
• Principled Training of Neural Networks with Direct Feedback Alignment
• Decoupled Greedy Learning of CNNs
• Low-Power Neuromorphic Hardware for Signal Processing Applications

Cited by in corpus (16)

• Bottom-up and top-down approaches for the design of neuromorphic processing systems: Tradeoffs and synergies between natural and artificial intelligence
• SoftHebb: Bayesian Inference in Unsupervised Hebbian Soft Winner-Take-All Networks
• ETLP: Event-based Three-factor Local Plasticity for online learning with neuromorphic hardware
• Convolutional Channel-wise Competitive Learning for the Forward-Forward Algorithm
• NeuroPack: An Algorithm-level Python-based Simulator for Memristor-empowered Neuro-inspired Computing
• Hardware Beyond Backpropagation: a Photonic Co-Processor for Direct Feedback Alignment
• Self-Contrastive Forward-Forward Algorithm
• Spiking Neural Predictive Coding for Continual Learning from Data Streams
• LLS: Local Learning Rule for Deep Neural Networks Inspired by Neural Activity Synchronization
• Photonic Differential Privacy with Direct Feedback Alignment
• Feed-Forward Optimization With Delayed Feedback for Neural Network Training
• XConv: Low-memory stochastic backpropagation for convolutional layers
• TESS: A Scalable Temporally and Spatially Local Learning Rule for Spiking Neural Networks
• Convergence Analysis and Implicit Regularization of Feedback Alignment for Deep Linear Networks
• Beyond Backpropagation: Optimization with Multi-Tangent Forward Gradients
• Training Multi-Layer Binary Neural Networks With Local Binary Error Signals