A Comprehensive Survey on Hardware-Aware Neural Architecture Search
arXiv:2101.09336
Abstract
Neural Architecture Search (NAS) methods have been growing in popularity. These techniques have been fundamental to automate and speed up the time consuming and error-prone process of synthesizing novel Deep Learning (DL) architectures. NAS has been extensively studied in the past few years. Arguably their most significant impact has been in image classification and object detection tasks where the state of the art results have been obtained. Despite the significant success achieved to date, applying NAS to real-world problems still poses significant challenges and is not widely practical. In general, the synthesized Convolution Neural Network (CNN) architectures are too complex to be deployed in resource-limited platforms, such as IoT, mobile, and embedded systems. One solution growing in popularity is to use multi-objective optimization algorithms in the NAS search strategy by taking into account execution latency, energy consumption, memory footprint, etc. This kind of NAS, called hardware-aware NAS (HW-NAS), makes searching the most efficient architecture more complicated and opens several questions. In this survey, we provide a detailed review of existing HW-NAS research and categorize them according to four key dimensions: the search space, the search strategy, the acceleration technique, and the hardware cost estimation strategies. We further discuss the challenges and limitations of existing approaches and potential future directions. This is the first survey paper focusing on hardware-aware NAS. We hope it serves as a valuable reference for the various techniques and algorithms discussed and paves the road for future research towards hardware-aware NAS.
arXiv commentSubmitted to Proceedings of IEEE
References in corpus (52)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Natural Language Processing (almost) from Scratch
- Categorical Reparameterization with Gumbel-Softmax
- DARTS: Differentiable Architecture Search
- Binarized Neural Networks
- A Survey of Model Compression and Acceleration for Deep Neural Networks
- To prune, or not to prune: exploring the efficacy of pruning for model compression
- Large-Scale Evolution of Image Classifiers
- The Microsoft 2016 Conversational Speech Recognition System
- NAS-Bench-101: Towards Reproducible Neural Architecture Search
- Single Path One-Shot Neural Architecture Search with Uniform Sampling
- Improved Noisy Student Training for Automatic Speech Recognition
- HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
- A Survey on Neural Architecture Search
- Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
- Neural Architecture Transfer
- NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size
- MicroNets: Neural Network Architectures for Deploying TinyML Applications on Commodity Microcontrollers
- MONAS: Multi-Objective Neural Architecture Search using Reinforcement Learning
- Fast, Accurate and Lightweight Super-Resolution with Neural Architecture Search
- Peephole: Predicting Network Performance Before Training
- NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
- NAT: Neural Architecture Transformer for Accurate and Compact Architectures
- The unreasonable effectiveness of the forget gate
- Long Short-Term Memory Over Tree Structures
- On Neural Architecture Search for Resource-Constrained Hardware Platforms
- Accelerator-aware Neural Network Design using AutoML
- Neural Architecture Search without Training
- MobileDets: Searching for Object Detection Architectures for Mobile Accelerators
- You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization
- Multi-Objective Reinforced Evolution in Mobile Neural Architecture Search
- Latency-Aware Differentiable Neural Architecture Search
- Evolving Robust Neural Architectures to Defend from Adversarial Attacks
- Design Automation for Efficient Deep Learning Computing
- DNA: Differentiable Network-Accelerator Co-Search
- Finding Fast Transformers: One-Shot Neural Architecture Search by Component Composition
- Single-Path NAS: Device-Aware Efficient ConvNet Design
- Real-Time Execution of Large-scale Language Models on Mobile
- Standing on the Shoulders of Giants: Hardware and Neural Architecture Co-Search with Hot Start
- NASS: Optimizing Secure Inference via Neural Architecture Search
- Energy-Aware Neural Architecture Optimization with Fast Splitting Steepest Descent
- Neural Architecture Search for Deep Image Prior
- S3NAS: Fast NPU-aware Neural Architecture Search Methodology
- NeuNetS: An Automated Synthesis Engine for Neural Network Design
- Multi-objective Neural Architecture Search via Non-stationary Policy Gradient
- Adaptive Precision Training: Quantify Back Propagation in Neural Networks with Fixed-point Numbers
- Performance Prediction for Convolutional Neural Networks in Edge Devices
- Binarized Neural Architecture Search for Efficient Object Recognition
- PONAS: Progressive One-shot Neural Architecture Search for Very Efficient Deployment
- ENAS4D: Efficient Multi-stage CNN Architecture Search for Dynamic Inference