SPIDE: A Purely Spike-based Method for Training Feedback Spiking Neural Networks

Abstract

Spiking neural networks (SNNs) with event-based computation are promising brain-inspired models for energy-efficient applications on neuromorphic hardware. However, most supervised SNN training methods, such as conversion from artificial neural networks or direct training with surrogate gradients, require complex computation rather than spike-based operations of spiking neurons during training. In this paper, we study spike-based implicit differentiation on the equilibrium state (SPIDE) that extends the recently proposed training method, implicit differentiation on the equilibrium state (IDE), for supervised learning with purely spike-based computation, which demonstrates the potential for energy-efficient training of SNNs. Specifically, we introduce ternary spiking neuron couples and prove that implicit differentiation can be solved by spikes based on this design, so the whole training procedure, including both forward and backward passes, is made as event-driven spike computation, and weights are updated locally with two-stage average firing rates. Then we propose to modify the reset membrane potential to reduce the approximation error of spikes. With these key components, we can train SNNs with flexible structures in a small number of time steps and with firing sparsity during training, and the theoretical estimation of energy costs demonstrates the potential for high efficiency. Meanwhile, experiments show that even with these constraints, our trained models can still achieve competitive results on MNIST, CIFAR-10, CIFAR-100, and CIFAR10-DVS. Our code is available at https://github.com/pkuxmq/SPIDE-FSNN.

arXiv commentAccepted by Neural Networks

References in corpus (24)

• Deep Learning in Spiking Neural Networks
• Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks
• STDP-based spiking deep convolutional neural networks for object recognition
• SLAYER: Spike Layer Error Reassignment in Time
• High-Performance Large-Scale Image Recognition Without Normalization
• Deep Equilibrium Models
• Neuromorphic Deep Learning Machines
• Deep Residual Learning in Spiking Neural Networks
• Spiking Deep Networks with LIF Neurons
• Direct Feedback Alignment Provides Learning in Deep Neural Networks
• Temporal Spike Sequence Learning via Backpropagation for Deep Spiking Neural Networks
• Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting
• Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networks
• Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks
• Multiscale Deep Equilibrium Models
• Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagation
• Online Training Through Time for Spiking Neural Networks
• A Free Lunch From ANN: Towards Efficient, Accurate Spiking Neural Networks Calibration
• Characterizing signal propagation to close the performance gap in unnormalized ResNets
• Towards deep learning with spiking neurons in energy based models with contrastive Hebbian plasticity
• A Statistical Framework for Low-bitwidth Training of Deep Neural Networks
• SpikeGrad: An ANN-equivalent Computation Model for Implementing Backpropagation with Spikes
• Stabilizing Equilibrium Models by Jacobian Regularization
• Minimax Dynamics of Optimally Balanced Spiking Networks of Excitatory and Inhibitory Neurons