Surrogate Gradient Learning in Spiking Neural Networks

Abstract

Spiking neural networks are nature's versatile solution to fault-tolerant and energy efficient signal processing. To translate these benefits into hardware, a growing number of neuromorphic spiking neural network processors attempt to emulate biological neural networks. These developments have created an imminent need for methods and tools to enable such systems to solve real-world signal processing problems. Like conventional neural networks, spiking neural networks can be trained on real, domain specific data. However, their training requires overcoming a number of challenges linked to their binary and dynamical nature. This article elucidates step-by-step the problems typically encountered when training spiking neural networks, and guides the reader through the key concepts of synaptic plasticity and data-driven learning in the spiking setting. To that end, it gives an overview of existing approaches and provides an introduction to surrogate gradient methods, specifically, as a particularly flexible and efficient method to overcome the aforementioned challenges.

References in corpus (3)

• Biologically inspired alternatives to backpropagation through time for learning in recurrent neural nets
• Optimal Spike-Timing Dependent Plasticity for Precise Action Potential Firing
• Temporally Efficient Deep Learning with Spikes

Cited by in corpus (7)

• SpykeTorch: Efficient Simulation of Convolutional Spiking Neural Networks with at most one Spike per Neuron
• Embodied Neuromorphic Vision with Event-Driven Random Backpropagation
• Deep Spiking Neural Networks for Large Vocabulary Automatic Speech Recognition
• Towards Scalable, Efficient and Accurate Deep Spiking Neural Networks with Backward Residual Connections, Stochastic Softmax and Hybridization
• Explicitly Trained Spiking Sparsity in Spiking Neural Networks with Backpropagation
• Harnessing Slow Dynamics in Neuromorphic Computation
• Action Recognition Using Supervised Spiking Neural Networks