Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting

Abstract

Recently, brain-inspired spiking neuron networks (SNNs) have attracted widespread research interest because of their event-driven and energy-efficient characteristics. Still, it is difficult to efficiently train deep SNNs due to the non-differentiability of its activation function, which disables the typically used gradient descent approaches for traditional artificial neural networks (ANNs). Although the adoption of surrogate gradient (SG) formally allows for the back-propagation of losses, the discrete spiking mechanism actually differentiates the loss landscape of SNNs from that of ANNs, failing the surrogate gradient methods to achieve comparable accuracy as for ANNs. In this paper, we first analyze why the current direct training approach with surrogate gradient results in SNNs with poor generalizability. Then we introduce the temporal efficient training (TET) approach to compensate for the loss of momentum in the gradient descent with SG so that the training process can converge into flatter minima with better generalizability. Meanwhile, we demonstrate that TET improves the temporal scalability of SNN and induces a temporal inheritable training for acceleration. Our method consistently outperforms the SOTA on all reported mainstream datasets, including CIFAR-10/100 and ImageNet. Remarkably on DVS-CIFAR10, we obtained 83$\%$ top-1 accuracy, over 10$\%$ improvement compared to existing state of the art. Codes are available at \url{https://github.com/Gus-Lab/temporal_efficient_training}.

arXiv commentPublished as a conference paper at ICLR 2022

Cited by in corpus (7)

• STSC-SNN: Spatio-Temporal Synaptic Connection with Temporal Convolution and Attention for Spiking Neural Networks
• Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training
• Evolutionary Spiking Neural Networks: A Survey
• SPIDE: A Purely Spike-based Method for Training Feedback Spiking Neural Networks
• Automotive Object Detection via Learning Sparse Events by Spiking Neurons
• Efficient Deep Spiking Multi-Layer Perceptrons with Multiplication-Free Inference
• STCSNN: High energy efficiency spike-train level spiking neural networks with spatio-temporal conversion