Deep Residual Learning in Spiking Neural Networks

Abstract

Deep Spiking Neural Networks (SNNs) present optimization difficulties for gradient-based approaches due to discrete binary activation and complex spatial-temporal dynamics. Considering the huge success of ResNet in deep learning, it would be natural to train deep SNNs with residual learning. Previous Spiking ResNet mimics the standard residual block in ANNs and simply replaces ReLU activation layers with spiking neurons, which suffers the degradation problem and can hardly implement residual learning. In this paper, we propose the spike-element-wise (SEW) ResNet to realize residual learning in deep SNNs. We prove that the SEW ResNet can easily implement identity mapping and overcome the vanishing/exploding gradient problems of Spiking ResNet. We evaluate our SEW ResNet on ImageNet, DVS Gesture, and CIFAR10-DVS datasets, and show that SEW ResNet outperforms the state-of-the-art directly trained SNNs in both accuracy and time-steps. Moreover, SEW ResNet can achieve higher performance by simply adding more layers, providing a simple method to train deep SNNs. To our best knowledge, this is the first time that directly training deep SNNs with more than 100 layers becomes possible. Our codes are available at https://github.com/fangwei123456/Spike-Element-Wise-ResNet.

arXiv commentAccepted by Advances in Neural Information Processing Systems (NeurIPS) 2021

References in corpus (21)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Deep Learning in Spiking Neural Networks
• Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks
• Mixed Precision Training
• One weird trick for parallelizing convolutional neural networks
• On the Number of Linear Regions of Deep Neural Networks
• SLAYER: Spike Layer Error Reassignment in Time
• Enabling Spike-based Backpropagation for Training Deep Neural Network Architectures
• Spiking Deep Networks with LIF Neurons
• Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networks
• DIET-SNN: Direct Input Encoding With Leakage and Threshold Optimization in Deep Spiking Neural Networks
• Spiking Deep Residual Network
• Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagation
• A Free Lunch From ANN: Towards Efficient, Accurate Spiking Neural Networks Calibration
• Technical report: supervised training of convolutional spiking neural networks with PyTorch
• Temporal-Coded Deep Spiking Neural Network with Easy Training and Robust Performance
• Going Deeper With Directly-Trained Larger Spiking Neural Networks
• Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks

Cited by in corpus (12)

• StereoSpike: Depth Learning with a Spiking Neural Network
• Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training
• Keys to Accurate Feature Extraction Using Residual Spiking Neural Networks
• Learning Delays in Spiking Neural Networks using Dilated Convolutions with Learnable Spacings
• SPIDE: A Purely Spike-based Method for Training Feedback Spiking Neural Networks
• Automotive Object Detection via Learning Sparse Events by Spiking Neurons
• Differentiable architecture search with multi-dimensional attention for spiking neural networks
• Efficient Deep Spiking Multi-Layer Perceptrons with Multiplication-Free Inference
• Optimising Event-Driven Spiking Neural Network with Regularisation and Cutoff
• Attacking the Spike: On the Transferability and Security of Spiking Neural Networks to Adversarial Examples
• S3TC: Spiking Separated Spatial and Temporal Convolutions with Unsupervised STDP-based Learning for Action Recognition
• Consistent model selection for estimating functional interactions among stochastic neurons with variable-length memory