Training Sparse Neural Networks using Compressed Sensing

Abstract

Pruning the weights of neural networks is an effective and widely-used technique for reducing model size and inference complexity. We develop and test a novel method based on compressed sensing which combines the pruning and training into a single step. Specifically, we utilize an adaptively weighted ${\ell }^1$ penalty on the weights during training, which we combine with a generalization of the regularized dual averaging (RDA) algorithm in order to train sparse neural networks. The adaptive weighting we introduce corresponds to a novel regularizer based on the logarithm of the absolute value of the weights. We perform a series of ablation studies demonstrating the improvement provided by the adaptive weighting and generalized RDA algorithm. Furthermore, numerical experiments on the CIFAR-10, CIFAR-100, and ImageNet datasets demonstrate that our method 1) trains sparser, more accurate networks than existing state-of-the-art methods; 2) can be used to train sparse networks from scratch, i.e. from a random initialization, as opposed to initializing with a well-trained base model; 3) acts as an effective regularizer, improving generalization accuracy.

References in corpus (23)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Understanding deep learning requires rethinking generalization
• Rethinking the Value of Network Pruning
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Pruning Filters for Efficient ConvNets
• Bayesian Compression for Deep Learning
• Variational Dropout Sparsifies Deep Neural Networks
• Training and Inference with Integers in Deep Neural Networks
• Learning the Number of Neurons in Deep Networks
• Sparse Networks from Scratch: Faster Training without Losing Performance
• Compression-aware Training of Deep Networks
• Global Sparse Momentum SGD for Pruning Very Deep Neural Networks
• Rigging the Lottery: Making All Tickets Winners
• Diversity Networks: Neural Network Compression Using Determinantal Point Processes
• Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers
• Fast ConvNets Using Group-wise Brain Damage