Exploring the Connection Between Binary and Spiking Neural Networks

Abstract

On-chip edge intelligence has necessitated the exploration of algorithmic techniques to reduce the compute requirements of current machine learning frameworks. This work aims to bridge the recent algorithmic progress in training Binary Neural Networks and Spiking Neural Networks - both of which are driven by the same motivation and yet synergies between the two have not been fully explored. We show that training Spiking Neural Networks in the extreme quantization regime results in near full precision accuracies on large-scale datasets like CIFAR-$100$ and ImageNet. An important implication of this work is that Binary Spiking Neural Networks can be enabled by "In-Memory" hardware accelerators catered for Binary Neural Networks without suffering any accuracy degradation due to binarization. We utilize standard training techniques for non-spiking networks to generate our spiking networks by conversion process and also perform an extensive empirical analysis and explore simple design-time and run-time optimization techniques for reducing inference latency of spiking networks (both for binary and full-precision models) by an order of magnitude over prior work.

References in corpus (14)

• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Ternary Weight Networks
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• BindsNET: A machine learning-oriented spiking neural networks library in Python
• Proposal for an All-Spin Artificial Neural Network: Emulating Neural and Synaptic Functionalities Through Domain Wall Motion in Ferromagnets
• Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
• High-Throughput In-Memory Computing for Binary Deep Neural Networks with Monolithically Integrated RRAM and 90nm CMOS
• Compression-aware Training of Deep Networks
• Constructing Energy-efficient Mixed-precision Neural Networks through Principal Component Analysis for Edge Intelligence
• Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagation
• Minibatch Processing in Spiking Neural Networks

Cited by in corpus (12)

• DIET-SNN: Direct Input Encoding With Leakage and Threshold Optimization in Deep Spiking Neural Networks
• BS4NN: Binarized Spiking Neural Networks with Temporal Coding and Learning
• TCL: an ANN-to-SNN Conversion with Trainable Clipping Layers
• SNN2ANN: A Fast and Memory-Efficient Training Framework for Spiking Neural Networks
• A Little Energy Goes a Long Way: Build an Energy-Efficient, Accurate Spiking Neural Network from Convolutional Neural Network
• Prominent characteristics of recurrent neuronal networks are robust against low synaptic weight resolution
• Towards Low-Latency Energy-Efficient Deep SNNs via Attention-Guided Compression
• DCT-SNN: Using DCT to Distribute Spatial Information over Time for Learning Low-Latency Spiking Neural Networks
• A Bop and Beyond: A Second Order Optimizer for Binarized Neural Networks
• One Timestep is All You Need: Training Spiking Neural Networks with Ultra Low Latency
• Neuromorphic Processing and Sensing: Evolutionary Progression of AI to Spiking
• Attacking the Spike: On the Transferability and Security of Spiking Neural Networks to Adversarial Examples