Single Shot Structured Pruning Before Training
arXiv:2007.00389
Abstract
We introduce a method to speed up training by 2x and inference by 3x in deep neural networks using structured pruning applied before training. Unlike previous works on pruning before training which prune individual weights, our work develops a methodology to remove entire channels and hidden units with the explicit aim of speeding up training and inference. We introduce a compute-aware scoring mechanism which enables pruning in units of sensitivity per FLOP removed, allowing even greater speed ups. Our method is fast, easy to implement, and needs just one forward/backward pass on a single batch of data to complete pruning before training begins.
References in corpus (7)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Distilling the Knowledge in a Neural Network
- AMC: AutoML for Model Compression and Acceleration on Mobile Devices
- Compressing Deep Convolutional Networks using Vector Quantization
- Picking Winning Tickets Before Training by Preserving Gradient Flow
- Faster gaze prediction with dense networks and Fisher pruning
- EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis
Cited by in corpus (5)
- MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge
- Adaptive Dense-to-Sparse Paradigm for Pruning Online Recommendation System with Non-Stationary Data
- Effective Model Sparsification by Scheduled Grow-and-Prune Methods
- SOSP: Efficiently Capturing Global Correlations by Second-Order Structured Pruning
- Statistical Mechanical Analysis of Neural Network Pruning