HALP: Hardware-Aware Latency Pruning

Abstract

Structural pruning can simplify network architecture and improve inference speed. We propose Hardware-Aware Latency Pruning (HALP) that formulates structural pruning as a global resource allocation optimization problem, aiming at maximizing the accuracy while constraining latency under a predefined budget. For filter importance ranking, HALP leverages latency lookup table to track latency reduction potential and global saliency score to gauge accuracy drop. Both metrics can be evaluated very efficiently during pruning, allowing us to reformulate global structural pruning under a reward maximization problem given target constraint. This makes the problem solvable via our augmented knapsack solver, enabling HALP to surpass prior work in pruning efficacy and accuracy-efficiency trade-off. We examine HALP on both classification and detection tasks, over varying networks, on ImageNet and VOC datasets. In particular, for ResNet-50/-101 pruning on ImageNet, HALP improves network throughput by $1.60\times$/$1.90\times$ with $+0.3\%$/$-0.2\%$ top-1 accuracy changes, respectively. For SSD pruning on VOC, HALP improves throughput by $1.94\times$ with only a $0.56$ mAP drop. HALP consistently outperforms prior art, sometimes by large margins.

References in corpus (18)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Language Models are Few-Shot Learners
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• cuDNN: Efficient Primitives for Deep Learning
• Rethinking the Value of Network Pruning
• Trained Ternary Quantization
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• The State of Sparsity in Deep Neural Networks
• Learning the Number of Neurons in Deep Networks
• AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
• Picking Winning Tickets Before Training by Preserving Gradient Flow
• Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks
• Progressive Skeletonization: Trimming more fat from a network at initialization
• APQ: Joint Search for Network Architecture, Pruning and Quantization Policy