Neuromorphic Deep Learning Machines

Abstract

An ongoing challenge in neuromorphic computing is to devise general and computationally efficient models of inference and learning which are compatible with the spatial and temporal constraints of the brain. One increasingly popular and successful approach is to take inspiration from inference and learning algorithms used in deep neural networks. However, the workhorse of deep learning, the gradient descent Back Propagation (BP) rule, often relies on the immediate availability of network-wide information stored with high-precision memory, and precise operations that are difficult to realize in neuromorphic hardware. Remarkably, recent work showed that exact backpropagated weights are not essential for learning deep representations. Random BP replaces feedback weights with random ones and encourages the network to adjust its feed-forward weights to learn pseudo-inverses of the (random) feedback weights. Building on these results, we demonstrate an event-driven random BP (eRBP) rule that uses an error-modulated synaptic plasticity for learning deep representations in neuromorphic computing hardware. The rule requires only one addition and two comparisons for each synaptic weight using a two-compartment leaky Integrate & Fire (I&F) neuron, making it very suitable for implementation in digital or mixed-signal neuromorphic hardware. Our results show that using eRBP, deep representations are rapidly learned, achieving nearly identical classification accuracies compared to artificial neural network simulations on GPUs, while being robust to neural and synaptic state quantizations during learning.

References in corpus (19)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Improving neural networks by preventing co-adaptation of feature detectors
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing
• Neuromorphic electronic circuits for building autonomous cognitive systems
• EMNIST: an extension of MNIST to handwritten letters
• Neuromorphic Deep Learning Machines
• Event-Driven Contrastive Divergence for Spiking Neuromorphic Systems
• Spiking Deep Networks with LIF Neurons
• Direct Feedback Alignment Provides Learning in Deep Neural Networks
• Stochastic Synapses Enable Efficient Brain-Inspired Learning Machines
• Stochastic inference with spiking neurons in the high-conductance state
• Rounding Methods for Neural Networks with Low Resolution Synaptic Weights
• Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes
• Understanding Synthetic Gradients and Decoupled Neural Interfaces
• Energy-efficient neuromorphic classifiers
• Neural Sampling by Irregular Gating Inhibition of Spiking Neurons and Attractor Networks

Cited by in corpus (32)

• Event-based Vision: A Survey
• Deep Learning in Spiking Neural Networks
• Enabling Spike-based Backpropagation for Training Deep Neural Network Architectures
• Synaptic Plasticity Dynamics for Deep Continuous Local Learning (DECOLLE)
• The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks
• Neuromorphic Deep Learning Machines
• S4NN: temporal backpropagation for spiking neural networks with one spike per neuron
• Bio-inspired digit recognition using reward-modulated spike-timing-dependent plasticity in deep convolutional networks
• MorphIC: A 65-nm 738k-Synapse/mm${}^2$ Quad-Core Binary-Weight Digital Neuromorphic Processor with Stochastic Spike-Driven Online Learning
• Adaptive Extreme Edge Computing for Wearable Devices
• Biologically plausible deep learning -- but how far can we go with shallow networks?
• Event-Based Backpropagation can compute Exact Gradients for Spiking Neural Networks
• Bottom-up and top-down approaches for the design of neuromorphic processing systems: Tradeoffs and synergies between natural and artificial intelligence
• BS4NN: Binarized Spiking Neural Networks with Temporal Coding and Learning
• Learning without feedback: Fixed random learning signals allow for feedforward training of deep neural networks
• The importance of space and time in neuromorphic cognitive agents
• Adaptive Learning Rule for Hardware-based Deep Neural Networks Using Electronic Synapse Devices
• Hardware Implementation of Spiking Neural Networks Using Time-To-First-Spike Encoding
• ETLP: Event-based Three-factor Local Plasticity for online learning with neuromorphic hardware
• Dendritic-Inspired Processing Enables Bio-Plausible STDP in Compound Binary Synapses
• SPIDE: A Purely Spike-based Method for Training Feedback Spiking Neural Networks
• Paired Competing Neurons Improving STDP Supervised Local Learning In Spiking Neural Networks
• Self-Contrastive Forward-Forward Algorithm
• Spike Talk: Genesis and Neural Coding Scheme Translations
• Supervised training of spiking neural networks for robust deployment on mixed-signal neuromorphic processors
• Event-based update of synapses in voltage-based learning rules
• Implementing a foveal-pit inspired filter in a Spiking Convolutional Neural Network: a preliminary study
• EventNet: Asynchronous Recursive Event Processing
• Energy-Efficient CMOS Memristive Synapses for Mixed-Signal Neuromorphic System-on-a-Chip
• Inherent Weight Normalization in Stochastic Neural Networks
• Foveal-pit inspired filtering of DVS spike response
• SPA: Stochastic Probability Adjustment for System Balance of Unsupervised SNNs