A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions

Abstract

Deep learning has made breakthroughs and substantial in many fields due to its powerful automatic representation capabilities. It has been proven that neural architecture design is crucial to the feature representation of data and the final performance. However, the design of the neural architecture heavily relies on the researchers' prior knowledge and experience. And due to the limitations of human' inherent knowledge, it is difficult for people to jump out of their original thinking paradigm and design an optimal model. Therefore, an intuitive idea would be to reduce human intervention as much as possible and let the algorithm automatically design the neural architecture. Neural Architecture Search (NAS) is just such a revolutionary algorithm, and the related research work is complicated and rich. Therefore, a comprehensive and systematic survey on the NAS is essential. Previously related surveys have begun to classify existing work mainly based on the key components of NAS: search space, search strategy, and evaluation strategy. While this classification method is more intuitive, it is difficult for readers to grasp the challenges and the landmark work involved. Therefore, in this survey, we provide a new perspective: beginning with an overview of the characteristics of the earliest NAS algorithms, summarizing the problems in these early NAS algorithms, and then providing solutions for subsequent related research work. Besides, we conduct a detailed and comprehensive analysis, comparison, and summary of these works. Finally, we provide some possible future research directions.

arXiv commentAccepted by ACM Computing Surveys 2021

References in corpus (59)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Conditional Generative Adversarial Nets
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• mixup: Beyond Empirical Risk Minimization
• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• Wide Residual Networks
• Neural Architecture Search: A Survey
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• Deep Subdomain Adaptation Network for Image Classification
• FractalNet: Ultra-Deep Neural Networks without Residuals
• Efficient Neural Architecture Search via Parameter Sharing
• Large-Scale Evolution of Image Classifiers
• Neural Architecture Optimization
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• Searching for Efficient Multi-Scale Architectures for Dense Image Prediction
• ConvNet Architecture Search for Spatiotemporal Feature Learning
• Efficient Architecture Search by Network Transformation
• Shake-Shake regularization
• SNAS: Stochastic Neural Architecture Search
• Understanding and Robustifying Differentiable Architecture Search
• Random Search and Reproducibility for Neural Architecture Search
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• Hierarchical Neural Architecture Search for Deep Stereo Matching
• Hierarchical Representations for Efficient Architecture Search
• One-Shot Neural Architecture Search via Self-Evaluated Template Network
• A Survey on Neural Architecture Search
• Simple And Efficient Architecture Search for Convolutional Neural Networks
• DetNAS: Backbone Search for Object Detection
• Convolutional Neural Fabrics
• NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size
• Net2Net: Accelerating Learning via Knowledge Transfer
• Path-Level Network Transformation for Efficient Architecture Search
• N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning
• Graph HyperNetworks for Neural Architecture Search
• NAS evaluation is frustratingly hard
• Peephole: Predicting Network Performance Before Training
• SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers
• NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
• Probabilistic Neural Architecture Search
• MaskConnect: Connectivity Learning by Gradient Descent
• Efficient Neural Architecture Transformation Searchin Channel-Level for Object Detection
• Dual Path Networks
• Computation Reallocation for Object Detection
• Blockwisely Supervised Neural Architecture Search with Knowledge Distillation
• AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures
• Connectivity Learning in Multi-Branch Networks
• FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining
• Deep Pyramidal Residual Networks with Separated Stochastic Depth
• When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attacks
• AutoSpeech: Neural Architecture Search for Speaker Recognition
• Learnable Embedding Space for Efficient Neural Architecture Compression
• SGAS: Sequential Greedy Architecture Search
• Understanding Architectures Learnt by Cell-based Neural Architecture Search
• Improving One-shot NAS by Suppressing the Posterior Fading

Cited by in corpus (22)

• AutoML: A Survey of the State-of-the-Art
• A Comprehensive Survey of Scene Graphs: Generation and Application
• Differentiable Quantum Architecture Search
• Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey
• Quantum Embedding Search for Quantum Machine Learning
• Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?
• Residual-based error correction for neural operator accelerated infinite-dimensional Bayesian inverse problems
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• One Proxy Device Is Enough for Hardware-Aware Neural Architecture Search
• AdaSpring: Context-adaptive and Runtime-evolutionary Deep Model Compression for Mobile Applications
• A General Method For Automatic Discovery of Powerful Interactions In Click-Through Rate Prediction
• Less is More: Proxy Datasets in NAS approaches
• Thief, Beware of What Get You There: Towards Understanding Model Extraction Attack
• POPNASv2: An Efficient Multi-Objective Neural Architecture Search Technique
• Integrating Large Circular Kernels into CNNs through Neural Architecture Search
• AutoLoss-Zero: Searching Loss Functions from Scratch for Generic Tasks
• Towards Efficient Convolutional Network Models with Filter Distribution Templates
• Contrastive Embeddings for Neural Architectures
• BaLeNAS: Differentiable Architecture Search via the Bayesian Learning Rule
• GradFreeBits: Gradient Free Bit Allocation for Dynamic Low Precision Neural Networks
• LV-BERT: Exploiting Layer Variety for BERT
• Differentiable Architecture Pruning for Transfer Learning