SpikeGrad: An ANN-equivalent Computation Model for Implementing Backpropagation with Spikes
arXiv:1906.00851
Abstract
Event-based neuromorphic systems promise to reduce the energy consumption of deep learning tasks by replacing expensive floating point operations on dense matrices by low power sparse and asynchronous operations on spike events. While these systems can be trained increasingly well using approximations of the back-propagation algorithm, these implementations usually require high precision errors for training and are therefore incompatible with the typical communication infrastructure of neuromorphic circuits. In this work, we analyze how the gradient can be discretized into spike events when training a spiking neural network. To accelerate our simulation, we show that using a special implementation of the integrate-and-fire neuron allows us to describe the accumulated activations and errors of the spiking neural network in terms of an equivalent artificial neural network, allowing us to largely speed up training compared to an explicit simulation of all spike events. This way we are able to demonstrate that even for deep networks, the gradients can be discretized sufficiently well with spikes if the gradient is properly rescaled. This form of spike-based backpropagation enables us to achieve equivalent or better accuracies on the MNIST and CIFAR10 dataset than comparable state-of-the-art spiking neural networks trained with full precision gradients. The algorithm, which we call SpikeGrad, is based on accumulation and comparison operations and can naturally exploit sparsity in the gradient computation, which makes it an interesting choice for a spiking neuromorphic systems with on-chip learning capacities.
References in corpus (11)
- DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
- Deep Learning in Spiking Neural Networks
- Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks
- Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing
- Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures
- Whetstone: A Method for Training Deep Artificial Neural Networks for Binary Communication
- A Theory of Local Learning, the Learning Channel, and the Optimality of Backpropagation
- Deep Spiking Networks
- Deep Spiking Neural Network with Spike Count based Learning Rule
- Algorithm and Hardware Design of Discrete-Time Spiking Neural Networks Based on Back Propagation with Binary Activations
- Deep counter networks for asynchronous event-based processing