Weight Pruning via Adaptive Sparsity Loss
arXiv:2006.02768
Abstract
Pruning neural networks has regained interest in recent years as a means to compress state-of-the-art deep neural networks and enable their deployment on resource-constrained devices. In this paper, we propose a robust compressive learning framework that efficiently prunes network parameters during training with minimal computational overhead. We incorporate fast mechanisms to prune individual layers and build upon these to automatically prune the entire network under a user-defined budget constraint. Key to our end-to-end network pruning approach is the formulation of an intuitive and easy-to-implement adaptive sparsity loss that is used to explicitly control sparsity during training, enabling efficient budget-aware optimization. Extensive experiments demonstrate the effectiveness of the proposed framework for image classification on the CIFAR and ImageNet datasets using different architectures, including AlexNet, ResNets and Wide ResNets.
References in corpus (16)
- Distilling the Knowledge in a Neural Network
- Sequence to Sequence Learning with Neural Networks
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Attention-Based Models for Speech Recognition
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
- Binarized Neural Networks
- FractalNet: Ultra-Deep Neural Networks without Residuals
- A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers
- Variational Dropout Sparsifies Deep Neural Networks
- Moonshine: Distilling with Cheap Convolutions
- Learning Sparse Networks Using Targeted Dropout
- Hybrid Pruning: Thinner Sparse Networks for Fast Inference on Edge Devices