SpikePipe: Accelerated Training of Spiking Neural Networks via Inter-Layer Pipelining and Multiprocessor Scheduling
arXiv:2406.06879 · doi:10.1109/TCASAI.2024.3496837
Abstract
Spiking Neural Networks (SNNs) have gained popularity due to their high energy efficiency. Prior works have proposed various methods for training SNNs, including backpropagation-based methods. Training SNNs is computationally expensive compared to their conventional counterparts and would benefit from multiprocessor hardware acceleration. This is the first paper to propose inter-layer pipelining to accelerate training in SNNs using systolic array-based processors and multiprocessor scheduling. The impact of training using delayed gradients is observed using three networks training on different datasets, showing no degradation for small networks and < 10% degradation for large networks. The mapping of various training tasks of the SNN onto systolic arrays is formulated, and the proposed scheduling method is evaluated on the three networks. The results are compared against standard pipelining algorithms. The results show that the proposed method achieves an average speedup of 1.6X compared to standard pipelining algorithms, with an upwards of 2X improvement in some cases. The incurred communication overhead due to the proposed method is less than 0.5% of the total required communication of training.
References in corpus (7)
- Adam: A Method for Stochastic Optimization
- Deep Learning in Spiking Neural Networks
- Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks
- Enabling Spike-based Backpropagation for Training Deep Neural Network Architectures
- The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks
- FPDeep: Scalable Acceleration of CNN Training on Deeply-Pipelined FPGA Clusters
- LayerPipe: Accelerating Deep Neural Network Training by Intra-Layer and Inter-Layer Gradient Pipelining and Multiprocessor Scheduling