Enabling Spike-based Backpropagation for Training Deep Neural Network Architectures

Abstract

Spiking Neural Networks (SNNs) have recently emerged as a prominent neural computing paradigm. However, the typical shallow SNN architectures have limited capacity for expressing complex representations while training deep SNNs using input spikes has not been successful so far. Diverse methods have been proposed to get around this issue such as converting off-the-shelf trained deep Artificial Neural Networks (ANNs) to SNNs. However, the ANN-SNN conversion scheme fails to capture the temporal dynamics of a spiking system. On the other hand, it is still a difficult problem to directly train deep SNNs using input spike events due to the discontinuous, non-differentiable nature of the spike generation function. To overcome this problem, we propose an approximate derivative method that accounts for the leaky behavior of LIF neurons. This method enables training deep convolutional SNNs directly (with input spike events) using spike-based backpropagation. Our experiments show the effectiveness of the proposed spike-based learning on deep networks (VGG and Residual architectures) by achieving the best classification accuracies in MNIST, SVHN and CIFAR-10 datasets compared to other SNNs trained with a spike-based learning. Moreover, we analyze sparse event-based computations to demonstrate the efficacy of the proposed SNN training method for inference operation in the spiking domain.

Chankyu Lee and Syed Shakib Sarwar contributed equally to the work

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks
• Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing
• STDP-based spiking deep convolutional neural networks for object recognition
• Fixed Point Quantization of Deep Convolutional Networks
• SLAYER: Spike Layer Error Reassignment in Time
• Enabling Spike-based Backpropagation for Training Deep Neural Network Architectures
• Neuromorphic Deep Learning Machines
• Spiking Deep Networks with LIF Neurons
• Network Plasticity as Bayesian Inference
• Direct Training for Spiking Neural Networks: Faster, Larger, Better
• Bio-Inspired Spiking Convolutional Neural Network using Layer-wise Sparse Coding and STDP Learning

Cited by in corpus (50)

• Enabling Spike-based Backpropagation for Training Deep Neural Network Architectures
• The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks
• Deep Residual Learning in Spiking Neural Networks
• DIET-SNN: Direct Input Encoding With Leakage and Threshold Optimization in Deep Spiking Neural Networks
• Is Neuromorphic MNIST neuromorphic? Analyzing the discriminative power of neuromorphic datasets in the time domain
• TCL: an ANN-to-SNN Conversion with Trainable Clipping Layers
• Pruning of Deep Spiking Neural Networks through Gradient Rewiring
• Bilayer-skyrmion-based design of neuron and synapse for spiking neural network
• Efficient Hardware Acceleration of Sparsely Active Convolutional Spiking Neural Networks
• Spiking Neural Networks for Visual Place Recognition via Weighted Neuronal Assignments
• Expert System Gradient Descent Style Training: Development of a Defensible Artificial Intelligence Technique
• Keys to Accurate Feature Extraction Using Residual Spiking Neural Networks
• Training Deep Spiking Neural Networks
• Artificial Intelligence-based algorithms in medical image scan seg-mentation and intelligent visual-content generation -- a concise overview
• Integration of Leaky-Integrate-and-Fire-Neurons in Deep Learning Architectures
• Spiking neural networks trained via proxy
• Stochastic-Sign SGD for Federated Learning with Theoretical Guarantees
• A Little Energy Goes a Long Way: Build an Energy-Efficient, Accurate Spiking Neural Network from Convolutional Neural Network
• Applications of Spiking Neural Networks in Visual Place Recognition
• Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks
• Towards Low-Latency Energy-Efficient Deep SNNs via Attention-Guided Compression
• LIAF-Net: Leaky Integrate and Analog Fire Network for Lightweight and Efficient Spatiotemporal Information Processing
• A Homomorphic Encryption Framework for Privacy-Preserving Spiking Neural Networks
• Differentiable architecture search with multi-dimensional attention for spiking neural networks
• Fractional-order spike-timing-dependent gradient descent for multi-layer spiking neural networks
• DCT-SNN: Using DCT to Distribute Spatial Information over Time for Learning Low-Latency Spiking Neural Networks
• SpikePipe: Accelerated Training of Spiking Neural Networks via Inter-Layer Pipelining and Multiprocessor Scheduling
• SpikeNAS: A Fast Memory-Aware Neural Architecture Search Framework for Spiking Neural Network-based Embedded AI Systems
• Efficient Deep Spiking Multi-Layer Perceptrons with Multiplication-Free Inference
• A Study On the Effects of Pre-processing On Spatio-temporal Action Recognition Using Spiking Neural Networks Trained with STDP
• Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium State
• Deep Spiking Convolutional Neural Network for Single Object Localization Based On Deep Continuous Local Learning
• Strategy and Benchmark for Converting Deep Q-Networks to Event-Driven Spiking Neural Networks
• Training Energy-Efficient Deep Spiking Neural Networks with Time-to-First-Spike Coding
• HYPER-SNN: Towards Energy-efficient Quantized Deep Spiking Neural Networks for Hyperspectral Image Classification
• SpiKernel: A Kernel Size Exploration Methodology for Improving Accuracy of the Embedded Spiking Neural Network Systems
• Brain inspired neuronal silencing mechanism to enable reliable sequence identification
• A multi-agent evolutionary robotics framework to train spiking neural networks
• Tuning Convolutional Spiking Neural Network with Biologically-plausible Reward Propagation
• One Timestep is All You Need: Training Spiking Neural Networks with Ultra Low Latency
• Training Deep Spiking Auto-encoders without Bursting or Dying Neurons through Regularization
• Deep Spiking Neural Networks with Resonate-and-Fire Neurons
• Optimized spiking neurons classify images with high accuracy through temporal coding with two spikes
• A Fully Spiking Hybrid Neural Network for Energy-Efficient Object Detection
• A temporally and spatially local spike-based backpropagation algorithm to enable training in hardware
• D-SELD: Dataset-Scalable Exemplar LCA-Decoder
• Gated Parametric Neuron for Spike-based Audio Recognition
• Explicitly Trained Spiking Sparsity in Spiking Neural Networks with Backpropagation
• Timing-Based Backpropagation in Spiking Neural Networks Without Single-Spike Restrictions
• HIRE-SNN: Harnessing the Inherent Robustness of Energy-Efficient Deep Spiking Neural Networks by Training with Crafted Input Noise