Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagation

Abstract

Spiking Neural Networks (SNNs) operate with asynchronous discrete events (or spikes) which can potentially lead to higher energy-efficiency in neuromorphic hardware implementations. Many works have shown that an SNN for inference can be formed by copying the weights from a trained Artificial Neural Network (ANN) and setting the firing threshold for each layer as the maximum input received in that layer. These type of converted SNNs require a large number of time steps to achieve competitive accuracy which diminishes the energy savings. The number of time steps can be reduced by training SNNs with spike-based backpropagation from scratch, but that is computationally expensive and slow. To address these challenges, we present a computationally-efficient training technique for deep SNNs. We propose a hybrid training methodology: 1) take a converted SNN and use its weights and thresholds as an initialization step for spike-based backpropagation, and 2) perform incremental spike-timing dependent backpropagation (STDB) on this carefully initialized network to obtain an SNN that converges within few epochs and requires fewer time steps for input processing. STDB is performed with a novel surrogate gradient function defined using neuron's spike time. The proposed training methodology converges in less than 20 epochs of spike-based backpropagation for most standard image classification datasets, thereby greatly reducing the training complexity compared to training SNNs from scratch. We perform experiments on CIFAR-10, CIFAR-100, and ImageNet datasets for both VGG and ResNet architectures. We achieve top-1 accuracy of 65.19% for ImageNet dataset on SNN with 250 time steps, which is 10X faster compared to converted SNNs with similar accuracy.

arXiv commentInternational Conference on Learning Representations (ICLR), 2020 https://openreview.net/forum?id=B1xSperKvH&noteId=B1xSperKvH

References in corpus (6)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks
• SLAYER: Spike Layer Error Reassignment in Time
• Spiking Deep Networks with LIF Neurons
• Surrogate Gradient Learning in Spiking Neural Networks
• Direct Training for Spiking Neural Networks: Faster, Larger, Better

Cited by in corpus (14)

• Deep Residual Learning in Spiking Neural Networks
• Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead
• Exploring the Connection Between Binary and Spiking Neural Networks
• Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networks
• DIET-SNN: Direct Input Encoding With Leakage and Threshold Optimization in Deep Spiking Neural Networks
• TCL: an ANN-to-SNN Conversion with Trainable Clipping Layers
• SPIDE: A Purely Spike-based Method for Training Feedback Spiking Neural Networks
• Connection Pruning for Deep Spiking Neural Networks with On-Chip Learning
• DCT-SNN: Using DCT to Distribute Spatial Information over Time for Learning Low-Latency Spiking Neural Networks
• Population-coding and Dynamic-neurons improved Spiking Actor Network for Reinforcement Learning
• Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium State
• One Timestep is All You Need: Training Spiking Neural Networks with Ultra Low Latency
• Optimized spiking neurons classify images with high accuracy through temporal coding with two spikes
• L4-Norm Weight Adjustments for Converted Spiking Neural Networks