Structured Directional Pruning via Perturbation Orthogonal Projection
arXiv:2107.05328
Abstract
Structured pruning is an effective compression technique to reduce the computation of neural networks, which is usually achieved by adding perturbations to reduce network parameters at the cost of slightly increasing training loss. A more reasonable approach is to find a sparse minimizer along the flat minimum valley found by optimizers, i.e. stochastic gradient descent, which keeps the training loss constant. To achieve this goal, we propose the structured directional pruning based on orthogonal projecting the perturbations onto the flat minimum valley. We also propose a fast solver sDprun and further prove that it achieves directional pruning asymptotically after sufficient training. Experiments using VGG-Net and ResNet on CIFAR-10 and CIFAR-100 datasets show that our method obtains the state-of-the-art pruned accuracy (i.e. 93.97% on VGG16, CIFAR-10 task) without retraining. Experiments using DNN, VGG-Net and WRN28X10 on MNIST, CIFAR-10 and CIFAR-100 datasets demonstrate our method performs structured directional pruning, reaching the same minimum valley as the optimizer.
References in corpus (10)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Reconciling modern machine learning practice and the bias-variance trade-off
- AMC: AutoML for Model Compression and Acceleration on Mobile Devices
- Gradient Descent Happens in a Tiny Subspace
- An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
- When is a Convolutional Filter Easy To Learn?
- Learning One-hidden-layer ReLU Networks via Gradient Descent
- A generalization of regularized dual averaging and its dynamics