Grow-Push-Prune: aligning deep discriminants for effective structural network compression
arXiv:2009.13716
Abstract
Most of today's popular deep architectures are hand-engineered to be generalists. However, this design procedure usually leads to massive redundant, useless, or even harmful features for specific tasks. Unnecessarily high complexities render deep nets impractical for many real-world applications, especially those without powerful GPU support. In this paper, we attempt to derive task-dependent compact models from a deep discriminant analysis perspective. We propose an iterative and proactive approach for classification tasks which alternates between (1) a pushing step, with an objective to simultaneously maximize class separation, penalize co-variances, and push deep discriminants into alignment with a compact set of neurons, and (2) a pruning step, which discards less useful or even interfering neurons. Deconvolution is adopted to reverse 'unimportant' filters' effects and recover useful contributing sources. A simple network growing strategy based on the basic Inception module is proposed for challenging tasks requiring larger capacity than what the base net can offer. Experiments on the MNIST, CIFAR10, and ImageNet datasets demonstrate our approach's efficacy. On ImageNet, by pushing and pruning our grown Inception-88 model, we achieve more accurate models than Inception nets generated during growing, residual nets, and popular compact nets at similar sizes. We also show that our grown Inception nets (without hard-coded dimension alignment) clearly outperform residual nets of similar complexities.
References in corpus (19)
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Neural Architecture Search with Reinforcement Learning
- Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
- DARTS: Differentiable Architecture Search
- Pruning Convolutional Neural Networks for Resource Efficient Inference
- AMC: AutoML for Model Compression and Acceleration on Mobile Devices
- Compressing Deep Convolutional Networks using Vector Quantization
- Pruning Filters for Efficient ConvNets
- Efficient Neural Architecture Search via Parameter Sharing
- Speeding up Convolutional Neural Networks with Low Rank Expansions
- Better Mixing via Deep Representations
- Learning Functions: When Is Deep Better Than Shallow
- Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networks
- Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods
- Structured Pruning of Deep Convolutional Neural Networks
- Task dependent Deep LDA pruning of neural networks
- Efficient Gender Classification Using a Deep LDA-Pruned Net
- Practical Block-wise Neural Network Architecture Generation
- Growing Efficient Deep Networks by Structured Continuous Sparsification