A Comprehensive Survey on Hardware-Aware Neural Architecture Search
arXiv:2101.09336
Abstract
Neural Architecture Search (NAS) methods have been growing in popularity. These techniques have been fundamental to automate and speed up the time consuming and error-prone process of synthesizing novel Deep Learning (DL) architectures. NAS has been extensively studied in the past few years. Arguably their most significant impact has been in image classification and object detection tasks where the state of the art results have been obtained. Despite the significant success achieved to date, applying NAS to real-world problems still poses significant challenges and is not widely practical. In general, the synthesized Convolution Neural Network (CNN) architectures are too complex to be deployed in resource-limited platforms, such as IoT, mobile, and embedded systems. One solution growing in popularity is to use multi-objective optimization algorithms in the NAS search strategy by taking into account execution latency, energy consumption, memory footprint, etc. This kind of NAS, called hardware-aware NAS (HW-NAS), makes searching the most efficient architecture more complicated and opens several questions. In this survey, we provide a detailed review of existing HW-NAS research and categorize them according to four key dimensions: the search space, the search strategy, the acceleration technique, and the hardware cost estimation strategies. We further discuss the challenges and limitations of existing approaches and potential future directions. This is the first survey paper focusing on hardware-aware NAS. We hope it serves as a valuable reference for the various techniques and algorithms discussed and paves the road for future research towards hardware-aware NAS.
Submitted to Proceedings of IEEE
References in corpus (24)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Natural Language Processing (almost) from Scratch
- To prune, or not to prune: exploring the efficacy of pruning for model compression
- NAS-Bench-101: Towards Reproducible Neural Architecture Search
- A Survey on Neural Architecture Search
- Peephole: Predicting Network Performance Before Training
- NAT: Neural Architecture Transformer for Accurate and Compact Architectures
- Long Short-Term Memory Over Tree Structures
- On Neural Architecture Search for Resource-Constrained Hardware Platforms
- Multi-Objective Reinforced Evolution in Mobile Neural Architecture Search
- Design Automation for Efficient Deep Learning Computing
- Finding Fast Transformers: One-Shot Neural Architecture Search by Component Composition
- Single-Path NAS: Device-Aware Efficient ConvNet Design
- Standing on the Shoulders of Giants: Hardware and Neural Architecture Co-Search with Hot Start
- Neural Architecture Search for Deep Image Prior
- NASS: Optimizing Secure Inference via Neural Architecture Search
- S3NAS: Fast NPU-aware Neural Architecture Search Methodology
- Multi-objective Neural Architecture Search via Non-stationary Policy Gradient
- NeuNetS: An Automated Synthesis Engine for Neural Network Design
- Adaptive Precision Training: Quantify Back Propagation in Neural Networks with Fixed-point Numbers
- PONAS: Progressive One-shot Neural Architecture Search for Very Efficient Deployment
- Binarized Neural Architecture Search for Efficient Object Recognition
- ENAS4D: Efficient Multi-stage CNN Architecture Search for Dynamic Inference