APQ: Joint Search for Network Architecture, Pruning and Quantization Policy

Abstract

We present APQ for efficient deep learning inference on resource-constrained hardware. Unlike previous methods that separately search the neural architecture, pruning policy, and quantization policy, we optimize them in a joint manner. To deal with the larger design space it brings, a promising approach is to train a quantization-aware accuracy predictor to quickly get the accuracy of the quantized model and feed it to the search engine to select the best fit. However, training this quantization-aware accuracy predictor requires collecting a large number of quantized <model, accuracy> pairs, which involves quantization-aware finetuning and thus is highly time-consuming. To tackle this challenge, we propose to transfer the knowledge from a full-precision (i.e., fp32) accuracy predictor to the quantization-aware (i.e., int8) accuracy predictor, which greatly improves the sample efficiency. Besides, collecting the dataset for the fp32 accuracy predictor only requires to evaluate neural networks without any training cost by sampling from a pretrained once-for-all network, which is highly efficient. Extensive experiments on ImageNet demonstrate the benefits of our joint optimization approach. With the same accuracy, APQ reduces the latency/energy by 2x/1.3x over MobileNetV2+HAQ. Compared to the separate optimization approach (ProxylessNAS+AMC+HAQ), APQ achieves 2.3% higher ImageNet accuracy while reducing orders of magnitude GPU hours and CO2 emission, pushing the frontier for green AI that is environmental-friendly. The code and video are publicly available.

Accepted by CVPR 2020

References in corpus (12)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• Trained Ternary Quantization
• Channel Pruning for Accelerating Very Deep Neural Networks
• Efficient Architecture Search by Network Transformation
• ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications
• Learning to Design Circuits
• Design Automation for Efficient Deep Learning Computing

Cited by in corpus (15)

• Bringing AI To Edge: From Deep Learning's Perspective
• Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• One Proxy Device Is Enough for Hardware-Aware Neural Architecture Search
• DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture
• Sharpness-aware Quantization for Deep Neural Networks
• HALP: Hardware-Aware Latency Pruning
• A Quantitative Perspective on Values of Domain Knowledge for Machine Learning
• When to Prune? A Policy towards Early Structural Pruning
• Auto-NBA: Efficient and Effective Search Over the Joint Space of Networks, Bitwidths, and Accelerators
• Scaling Up Deep Neural Network Optimization for Edge Inference
• SoFAr: Shortcut-based Fractal Architectures for Binary Convolutional Neural Networks
• T-AutoML: Automated Machine Learning for Lesion Segmentation using Transformers in 3D Medical Imaging
• Evolutionary Algorithms in Approximate Computing: A Survey
• GPNAS: A Neural Network Architecture Search Framework Based on Graphical Predictor