Surrogate Gradient Learning in Spiking Neural Networks

Abstract

Spiking neural networks are nature's versatile solution to fault-tolerant and energy efficient signal processing. To translate these benefits into hardware, a growing number of neuromorphic spiking neural network processors attempt to emulate biological neural networks. These developments have created an imminent need for methods and tools to enable such systems to solve real-world signal processing problems. Like conventional neural networks, spiking neural networks can be trained on real, domain specific data. However, their training requires overcoming a number of challenges linked to their binary and dynamical nature. This article elucidates step-by-step the problems typically encountered when training spiking neural networks, and guides the reader through the key concepts of synaptic plasticity and data-driven learning in the spiking setting. To that end, it gives an overview of existing approaches and provides an introduction to surrogate gradient methods, specifically, as a particularly flexible and efficient method to overcome the aforementioned challenges.

References in corpus (13)

• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Deep Learning in Spiking Neural Networks
• Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing
• SLAYER: Spike Layer Error Reassignment in Time
• Supervised Learning in Spiking Neural Networks with FORCE Training
• Spiking Deep Networks with LIF Neurons
• A Theory of Local Learning, the Learning Channel, and the Optimality of Backpropagation
• Biologically inspired alternatives to backpropagation through time for learning in recurrent neural nets
• Encoding Spike Patterns in Multilayer Spiking Neural Networks
• SMPLR: Deep SMPL reverse for 3D human pose and shape recovery
• Optimal Spike-Timing Dependent Plasticity for Precise Action Potential Firing
• Temporally Efficient Deep Learning with Spikes

Cited by in corpus (40)

• Synaptic Plasticity Dynamics for Deep Continuous Local Learning (DECOLLE)
• The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks
• SpykeTorch: Efficient Simulation of Convolutional Spiking Neural Networks with at most one Spike per Neuron
• Review: Deep Learning in Electron Microscopy
• DIET-SNN: Direct Input Encoding With Leakage and Threshold Optimization in Deep Spiking Neural Networks
• Federated Learning with Spiking Neural Networks
• BS4NN: Binarized Spiking Neural Networks with Temporal Coding and Learning
• TCL: an ANN-to-SNN Conversion with Trainable Clipping Layers
• Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagation
• RMP-SNN: Residual Membrane Potential Neuron for Enabling Deeper High-Accuracy and Low-Latency Spiking Neural Network
• Spiking neural networks trained with backpropagation for low power neuromorphic implementation of voice activity detection
• Integration of Leaky-Integrate-and-Fire-Neurons in Deep Learning Architectures
• Stochasticity and Robustness in Spiking Neural Networks
• Learning Delays in Spiking Neural Networks using Dilated Convolutions with Learnable Spacings
• Exploring Adversarial Attack in Spiking Neural Networks with Spike-Compatible Gradient
• A Tandem Learning Rule for Effective Training and Rapid Inference of Deep Spiking Neural Networks
• Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks
• Parallel Hyperparameter Optimization Of Spiking Neural Network
• Visual Pattern Recognition with on On-chip Learning: towards a Fully Neuromorphic Approach
• Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networks
• Fast and energy-efficient neuromorphic deep learning with first-spike times
• Exploiting Neuron and Synapse Filter Dynamics in Spatial Temporal Learning of Deep Spiking Neural Network
• Embodied Neuromorphic Vision with Event-Driven Random Backpropagation
• DCT-SNN: Using DCT to Distribute Spatial Information over Time for Learning Low-Latency Spiking Neural Networks
• Deep Spiking Neural Networks for Large Vocabulary Automatic Speech Recognition
• Towards Scalable, Efficient and Accurate Deep Spiking Neural Networks with Backward Residual Connections, Stochastic Softmax and Hybridization
• Supervised training of spiking neural networks for robust deployment on mixed-signal neuromorphic processors
• You Only Spike Once: Improving Energy-Efficient Neuromorphic Inference to ANN-Level Accuracy
• Beyond Classification: Directly Training Spiking Neural Networks for Semantic Segmentation
• Ensembles of Spiking Neural Networks
• One Timestep is All You Need: Training Spiking Neural Networks with Ultra Low Latency
• Spiking Neural Networks with Single-Spike Temporal-Coded Neurons for Network Intrusion Detection
• Minibatch Processing in Spiking Neural Networks
• Multivariate Time Series Classification Using Spiking Neural Networks
• Explicitly Trained Spiking Sparsity in Spiking Neural Networks with Backpropagation
• Harnessing Slow Dynamics in Neuromorphic Computation
• Fitting summary statistics of neural data with a differentiable spiking network simulator
• An Introduction to Spiking Neural Networks: Probabilistic Models, Learning Rules, and Applications
• Minimal spiking neuron for solving multi-label classification tasks
• Action Recognition Using Supervised Spiking Neural Networks