Knapsack Pruning with Inner Distillation

Abstract

Neural network pruning reduces the computational cost of an over-parameterized network to improve its efficiency. Popular methods vary from ${\ell }_1$-norm sparsification to Neural Architecture Search (NAS). In this work, we propose a novel pruning method that optimizes the final accuracy of the pruned network and distills knowledge from the over-parameterized parent network's inner layers. To enable this approach, we formulate the network pruning as a Knapsack Problem which optimizes the trade-off between the importance of neurons and their associated computational cost. Then we prune the network channels while maintaining the high-level structure of the network. The pruned network is fine-tuned under the supervision of the parent network using its inner network knowledge, a technique we refer to as the Inner Knowledge Distillation. Our method leads to state-of-the-art pruning results on ImageNet, CIFAR-10 and CIFAR-100 using ResNet backbones. To prune complex network structures such as convolutions with skip-links and depth-wise convolutions, we propose a block grouping approach to cope with these structures. Through this we produce compact architectures with the same FLOPs as EfficientNet-B0 and MobileNetV3 but with higher accuracy, by $1\%$ and $0.3\%$ respectively on ImageNet, and faster runtime on GPU.

References in corpus (7)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Global Sparse Momentum SGD for Pruning Very Deep Neural Networks
• Moonshine: Distilling with Cheap Convolutions

Cited by in corpus (5)

• Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programming
• HardCoRe-NAS: Hard Constrained diffeRentiable Neural Architecture Search
• The Role of Regularization in Shaping Weight and Node Pruning Dependency and Dynamics
• Measure Twice, Cut Once: Quantifying Bias and Fairness in Deep Neural Networks
• Exploiting Features with Split-and-Share Module