Comprehensive SNN Compression Using ADMM Optimization and Activity Regularization

Abstract

As well known, the huge memory and compute costs of both artificial neural networks (ANNs) and spiking neural networks (SNNs) greatly hinder their deployment on edge devices with high efficiency. Model compression has been proposed as a promising technique to improve the running efficiency via parameter and operation reduction. Whereas, this technique is mainly practiced in ANNs rather than SNNs. It is interesting to answer how much an SNN model can be compressed without compromising its functionality, where two challenges should be addressed: i) the accuracy of SNNs is usually sensitive to model compression, which requires an accurate compression methodology; ii) the computation of SNNs is event-driven rather than static, which produces an extra compression dimension on dynamic spikes. To this end, we realize a comprehensive SNN compression through three steps. First, we formulate the connection pruning and weight quantization as a constrained optimization problem. Second, we combine spatio-temporal backpropagation (STBP) and alternating direction method of multipliers (ADMM) to solve the problem with minimum accuracy loss. Third, we further propose activity regularization to reduce the spike events for fewer active operations. These methods can be applied in either a single way for moderate compression or a joint way for aggressive compression. We define several quantitative metrics to evaluation the compression performance for SNNs. Our methodology is validated in pattern recognition tasks over MNIST, N-MNIST, CIFAR10, and CIFAR100 datasets, where extensive comparisons, analyses, and insights are provided. To our best knowledge, this is the first work that studies SNN compression in a comprehensive manner by exploiting all compressible components and achieves better results.

Under review

References in corpus (15)

• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks
• Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing
• Tensorizing Neural Networks
• A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers
• Training and Inference with Integers in Deep Neural Networks
• NullHop: A Flexible Convolutional Neural Network Accelerator Based on Sparse Representations of Feature Maps
• Direct Training for Spiking Neural Networks: Faster, Larger, Better
• Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM
• Spiking Deep Residual Network
• Crossbar-aware neural network pruning
• Progressive Weight Pruning of Deep Neural Networks using ADMM
• StructADMM: A Systematic, High-Efficiency Framework of Structured Weight Pruning for DNNs

Cited by in corpus (2)

• Towards Low-Latency Energy-Efficient Deep SNNs via Attention-Guided Compression
• Connection Pruning for Deep Spiking Neural Networks with On-Chip Learning