Efficient Neural Architecture Search via Parameter Sharing

Abstract

We propose Efficient Neural Architecture Search (ENAS), a fast and inexpensive approach for automatic model design. In ENAS, a controller learns to discover neural network architectures by searching for an optimal subgraph within a large computational graph. The controller is trained with policy gradient to select a subgraph that maximizes the expected reward on the validation set. Meanwhile the model corresponding to the selected subgraph is trained to minimize a canonical cross entropy loss. Thanks to parameter sharing between child models, ENAS is fast: it delivers strong empirical performances using much fewer GPU-hours than all existing automatic model design approaches, and notably, 1000x less expensive than standard Neural Architecture Search. On the Penn Treebank dataset, ENAS discovers a novel architecture that achieves a test perplexity of 55.8, establishing a new state-of-the-art among all methods without post-training processing. On the CIFAR-10 dataset, ENAS designs novel architectures that achieve a test error of 2.89%, which is on par with NASNet (Zoph et al., 2018), whose test error is 2.65%.

References in corpus (10)

• Deep Residual Learning for Image Recognition
• Recurrent Neural Network Regularization
• Densely Connected Convolutional Networks
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Regularizing and Optimizing LSTM Language Models
• Efficient Architecture Search by Network Transformation
• Neural Optimizer Search with Reinforcement Learning
• Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
• Peephole: Predicting Network Performance Before Training
• Capacity and Trainability in Recurrent Neural Networks

Cited by in corpus (266)

• Transformers in Vision: A Survey
• AutoML: A Survey of the State-of-the-Art
• Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
• SNAS: Stochastic Neural Architecture Search
• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• Random Search and Reproducibility for Neural Architecture Search
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• Evaluating the Search Phase of Neural Architecture Search
• Differentiable Learning-to-Normalize via Switchable Normalization
• Path-Level Network Transformation for Efficient Architecture Search
• AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
• Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation
• FasterSeg: Searching for Faster Real-time Semantic Segmentation
• Benchmarking the Robustness of Semantic Segmentation Models
• Review of data analysis in vision inspection of power lines with an in-depth discussion of deep learning technology
• FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
• D-VAE: A Variational Autoencoder for Directed Acyclic Graphs
• Scalable Neural Architecture Search for 3D Medical Image Segmentation
• Auto-Keras: An Efficient Neural Architecture Search System
• NSGA-Net: Neural Architecture Search using Multi-Objective Genetic Algorithm
• Learning to Continually Learn
• BNAS:An Efficient Neural Architecture Search Approach Using Broad Scalable Architecture
• EEEA-Net: An Early Exit Evolutionary Neural Architecture Search
• MaskConnect: Connectivity Learning by Gradient Descent
• A Survey of Deep Learning for Scientific Discovery
• AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search
• Evolving Normalization-Activation Layers
• Real-time 3D Nanoscale Coherent Imaging via Physics-aware Deep Learning
• Efficient Neural Architecture Transformation Searchin Channel-Level for Object Detection
• 3D-ANAS: 3D Asymmetric Neural Architecture Search for Fast Hyperspectral Image Classification
• ReLeQ: A Reinforcement Learning Approach for Deep Quantization of Neural Networks
• Learning to Branch for Multi-Task Learning
• Auto-ReID: Searching for a Part-aware ConvNet for Person Re-Identification
• MobileDets: Searching for Object Detection Architectures for Mobile Accelerators
• Routing Networks and the Challenges of Modular and Compositional Computation
• BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models
• AGAN: Towards Automated Design of Generative Adversarial Networks
• Universality and individuality in neural dynamics across large populations of recurrent networks
• Multi-Objective Reinforced Evolution in Mobile Neural Architecture Search
• Adversarially Robust Neural Architectures
• Noisy Differentiable Architecture Search
• Contrastive Self-supervised Neural Architecture Search
• PHOTONAI -- A Python API for Rapid Machine Learning Model Development
• Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?
• Blending Diverse Physical Priors with Neural Networks
• Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing
• Neural Graph Evolution: Towards Efficient Automatic Robot Design
• CryptoNAS: Private Inference on a ReLU Budget
• Evaluating Efficient Performance Estimators of Neural Architectures
• Placeto: Learning Generalizable Device Placement Algorithms for Distributed Machine Learning
• DARTS-: Robustly Stepping out of Performance Collapse Without Indicators
• Budgeted Training: Rethinking Deep Neural Network Training Under Resource Constraints
• FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions
• AutoLoss: Learning Discrete Schedules for Alternate Optimization
• A Comprehensive Review and a Taxonomy of Edge Machine Learning: Requirements, Paradigms, and Techniques
• Learning Graph Convolutional Network for Skeleton-based Human Action Recognition by Neural Searching
• CVR-Net: A deep convolutional neural network for coronavirus recognition from chest radiography images
• DSNAS: Direct Neural Architecture Search without Parameter Retraining
• AutoHAS: Efficient Hyperparameter and Architecture Search
• Learning Implicitly Recurrent CNNs Through Parameter Sharing
• RC-DARTS: Resource Constrained Differentiable Architecture Search
• CARS: Continuous Evolution for Efficient Neural Architecture Search
• A Generic Graph-based Neural Architecture Encoding Scheme for Predictor-based NAS
• Quantitative Phase Imaging and Artificial Intelligence: A Review
• FEED: Feature-level Ensemble for Knowledge Distillation
• Federated Neural Architecture Search
• NAS-FCOS: Fast Neural Architecture Search for Object Detection
• DCNAS: Densely Connected Neural Architecture Search for Semantic Image Segmentation
• AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations
• DiNTS: Differentiable Neural Network Topology Search for 3D Medical Image Segmentation
• Progressive DARTS: Bridging the Optimization Gap for NAS in the Wild
• Neural Predictor for Neural Architecture Search
• Evolving Reinforcement Learning Algorithms
• Evolving Robust Neural Architectures to Defend from Adversarial Attacks
• HARK Side of Deep Learning -- From Grad Student Descent to Automated Machine Learning
• DPP-Net: Device-aware Progressive Search for Pareto-optimal Neural Architectures
• C2FNAS: Coarse-to-Fine Neural Architecture Search for 3D Medical Image Segmentation
• Memory-efficient Embedding for Recommendations
• A General Method For Automatic Discovery of Powerful Interactions In Click-Through Rate Prediction
• ReNAS:Relativistic Evaluation of Neural Architecture Search
• MoViNets: Mobile Video Networks for Efficient Video Recognition
• An Adaptive Random Path Selection Approach for Incremental Learning
• Hardware-Centric AutoML for Mixed-Precision Quantization
• Angle-based Search Space Shrinking for Neural Architecture Search
• Multinomial Distribution Learning for Effective Neural Architecture Search
• Multi-branch Attentive Transformer
• Evolutionary-Neural Hybrid Agents for Architecture Search
• Exploring Shared Structures and Hierarchies for Multiple NLP Tasks
• Towards NNGP-guided Neural Architecture Search
• GNAS: A Greedy Neural Architecture Search Method for Multi-Attribute Learning
• Evolutionary Spiking Neural Networks: A Survey
• AutoSpeech: Neural Architecture Search for Speaker Recognition
• Learnable Embedding Space for Efficient Neural Architecture Compression
• Reinforcement Learning and Adaptive Sampling for Optimized DNN Compilation
• AM-LFS: AutoML for Loss Function Search
• Efficient Bitwidth Search for Practical Mixed Precision Neural Network
• Orchestrating the Development Lifecycle of Machine Learning-Based IoT Applications: A Taxonomy and Survey
• AttentiveNAS: Improving Neural Architecture Search via Attentive Sampling
• Single-Path NAS: Device-Aware Efficient ConvNet Design
• BETANAS: BalancEd TrAining and selective drop for Neural Architecture Search
• SGAS: Sequential Greedy Architecture Search
• A Semi-Supervised Assessor of Neural Architectures
• Feature Partitioning for Efficient Multi-Task Architectures
• Deep Multimodal Neural Architecture Search
• Efficient Deep Neural Networks
• Sampled Training and Node Inheritance for Fast Evolutionary Neural Architecture Search
• Neural Architecture Search using Deep Neural Networks and Monte Carlo Tree Search
• Neural Architecture Search for Deep Face Recognition
• From Federated Learning to Federated Neural Architecture Search: A Survey
• Online Hyper-parameter Learning for Auto-Augmentation Strategy
• AutoCaption: Image Captioning with Neural Architecture Search
• Efficient Forward Architecture Search
• MnasFPN: Learning Latency-aware Pyramid Architecture for Object Detection on Mobile Devices
• DSRNA: Differentiable Search of Robust Neural Architectures
• Growing Efficient Deep Networks by Structured Continuous Sparsification
• TAPAS: Train-less Accuracy Predictor for Architecture Search
• Hierarchical Neural Architecture Search via Operator Clustering
• Neural Architecture Optimization with Graph VAE
• Understanding Architectures Learnt by Cell-based Neural Architecture Search
• MTL-NAS: Task-Agnostic Neural Architecture Search towards General-Purpose Multi-Task Learning
• Improving One-shot NAS by Suppressing the Posterior Fading
• IRLAS: Inverse Reinforcement Learning for Architecture Search
• Measuring Visual Generalization in Continuous Control from Pixels
• Neural Architecture Search For Fault Diagnosis
• Leveraging End-to-End Speech Recognition with Neural Architecture Search
• Automated Model Design and Benchmarking of 3D Deep Learning Models for COVID-19 Detection with Chest CT Scans
• CATE: Computation-aware Neural Architecture Encoding with Transformers
• Policy Optimization With Penalized Point Probability Distance: An Alternative To Proximal Policy Optimization
• Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics
• PV-NAS: Practical Neural Architecture Search for Video Recognition
• S3NAS: Fast NPU-aware Neural Architecture Search Methodology
• Self-Adaptive 2D-3D Ensemble of Fully Convolutional Networks for Medical Image Segmentation
• Effective and Fast: A Novel Sequential Single Path Search for Mixed-Precision Quantization
• Fast Neural Architecture Construction using EnvelopeNets
• Evolving Search Space for Neural Architecture Search
• AutoGrow: Automatic Layer Growing in Deep Convolutional Networks
• AgEBO-Tabular: Joint Neural Architecture and Hyperparameter Search with Autotuned Data-Parallel Training for Tabular Data
• AutoML to Date and Beyond: Challenges and Opportunities
• ScaleNAS: One-Shot Learning of Scale-Aware Representations for Visual Recognition
• VINNAS: Variational Inference-based Neural Network Architecture Search
• Do Normalization Layers in a Deep ConvNet Really Need to Be Distinct?
• Efficient Decoupled Neural Architecture Search by Structure and Operation Sampling
• On the Privacy Risks of Cell-Based NAS Architectures
• ModuleNet: Knowledge-inherited Neural Architecture Search
• HourNAS: Extremely Fast Neural Architecture Search Through an Hourglass Lens
• Policy-GNN: Aggregation Optimization for Graph Neural Networks
• NAS-Count: Counting-by-Density with Neural Architecture Search
• Self-Organizing Intelligent Matter: A blueprint for an AI generating algorithm
• DARC: Differentiable ARchitecture Compression
• Neural Architecture Search Over a Graph Search Space
• A Study on Encodings for Neural Architecture Search
• Genetic Neural Architecture Search for automatic assessment of human sperm images
• Resource Optimized Neural Architecture Search for 3D Medical Image Segmentation
• Latent Domain Learning with Dynamic Residual Adapters
• DC-NAS: Divide-and-Conquer Neural Architecture Search
• Neural Architecture Refinement: A Practical Way for Avoiding Overfitting in NAS
• Extractive Summary as Discrete Latent Variables
• Video Action Recognition Via Neural Architecture Searching
• Searching for Fast Model Families on Datacenter Accelerators
• Self-Assembling Modular Networks for Interpretable Multi-Hop Reasoning
• Intra-Ensemble in Neural Networks
• Trainless Model Performance Estimation for Neural Architecture Search
• Resource Constrained Neural Network Architecture Search: Will a Submodularity Assumption Help?
• AutoSTR: Efficient Backbone Search for Scene Text Recognition
• Fine-Grained Stochastic Architecture Search
• EDAS: Efficient and Differentiable Architecture Search
• TextNAS: A Neural Architecture Search Space tailored for Text Representation
• DAAS: Differentiable Architecture and Augmentation Policy Search
• Binarized Neural Architecture Search for Efficient Object Recognition
• Automatic Data Augmentation for 3D Medical Image Segmentation
• AutoReCon: Neural Architecture Search-based Reconstruction for Data-free Compression
• Full-attention based Neural Architecture Search using Context Auto-regression
• Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search
• Effective, Efficient and Robust Neural Architecture Search
• AIBench Training: Balanced Industry-Standard AI Training Benchmarking
• Neural Architecture Search for Gliomas Segmentation on Multimodal Magnetic Resonance Imaging
• On the Bounds of Function Approximations
• Differentiable Learning-to-Group Channels via Groupable Convolutional Neural Networks
• SCARLET-NAS: Bridging the Gap between Stability and Scalability in Weight-sharing Neural Architecture Search
• MemNet: Memory-Efficiency Guided Neural Architecture Search with Augment-Trim learning
• NASB: Neural Architecture Search for Binary Convolutional Neural Networks
• ACDC: Weight Sharing in Atom-Coefficient Decomposed Convolution
• Scheduled Differentiable Architecture Search for Visual Recognition
• Exploiting Operation Importance for Differentiable Neural Architecture Search
• ResNetX: a more disordered and deeper network architecture
• DNQ: Dynamic Network Quantization
• RandomNet: Towards Fully Automatic Neural Architecture Design for Multimodal Learning
• Regularize, Expand and Compress: Multi-task based Lifelong Learning via NonExpansive AutoML
• WeNet: Weighted Networks for Recurrent Network Architecture Search
• G-DARTS-A: Groups of Channel Parallel Sampling with Attention
• URNet : User-Resizable Residual Networks with Conditional Gating Module
• EnforceNet: Monocular Camera Localization in Large Scale Indoor Sparse LiDAR Point Cloud
• Efficient Semantic Segmentation using Gradual Grouping
• Regularized Evolutionary Population-Based Training
• Genetic Network Architecture Search
• Hypernetwork-Based Augmentation
• Interleaving Learning, with Application to Neural Architecture Search
• Analysis of an Automated Machine Learning Approach in Brain Predictive Modelling: A data-driven approach to Predict Brain Age from Cortical Anatomical Measures
• Representation Sharing for Fast Object Detector Search and Beyond
• Neural Architecture Search with Random Labels
• DA-NAS: Data Adapted Pruning for Efficient Neural Architecture Search
• Hyperparameter optimization with REINFORCE and Transformers
• HAO: Hardware-aware neural Architecture Optimization for Efficient Inference
• Ranking and Rejecting of Pre-Trained Deep Neural Networks in Transfer Learning based on Separation Index
• FP-NAS: Fast Probabilistic Neural Architecture Search
• GEVO: GPU Code Optimization using Evolutionary Computation
• DeepSwarm: Optimising Convolutional Neural Networks using Swarm Intelligence
• Automated Search for Configurations of Deep Neural Network Architectures
• HBONet: Harmonious Bottleneck on Two Orthogonal Dimensions
• Single-level Optimization For Differential Architecture Search
• Neural Rejuvenation: Improving Deep Network Training by Enhancing Computational Resource Utilization
• Efficient Human Pose Estimation with Depthwise Separable Convolution and Person Centroid Guided Joint Grouping
• An Efficient Quantitative Approach for Optimizing Convolutional Neural Networks
• Multi-path Neural Networks for On-device Multi-domain Visual Classification
• BN-NAS: Neural Architecture Search with Batch Normalization
• AutoADR: Automatic Model Design for Ad Relevance
• Improving the sample-efficiency of neural architecture search with reinforcement learning
• Evolutionary Algorithm Enhanced Neural Architecture Search for Text-Independent Speaker Verification
• BFTrainer: Low-Cost Training of Neural Networks on Unfillable Supercomputer Nodes
• Direct Federated Neural Architecture Search
• A hybrid quantum-classical neural network with deep residual learning
• FDNAS: Improving Data Privacy and Model Diversity in AutoML
• AutoShrink: A Topology-aware NAS for Discovering Efficient Neural Architecture
• EIGEN: Ecologically-Inspired GENetic Approach for Neural Network Structure Searching from Scratch
• Network Architecture Search for Face Enhancement
• MLFriend: Interactive Prediction Task Recommendation for Event-Driven Time-Series Data
• A Genetic Algorithm based Kernel-size Selection Approach for a Multi-column Convolutional Neural Network
• AutoPruning for Deep Neural Network with Dynamic Channel Masking
• ADWPNAS: Architecture-Driven Weight Prediction for Neural Architecture Search
• Compact CNN Structure Learning by Knowledge Distillation
• Pretraining Neural Architecture Search Controllers with Locality-based Self-Supervised Learning
• Network Adjustment: Channel Search Guided by FLOPs Utilization Ratio
• Unchain the Search Space with Hierarchical Differentiable Architecture Search
• Mutually-aware Sub-Graphs Differentiable Architecture Search
• Efficient Automatic Meta Optimization Search for Few-Shot Learning
• Graphs for deep learning representations
• Searching for TrioNet: Combining Convolution with Local and Global Self-Attention
• T-AutoML: Automated Machine Learning for Lesion Segmentation using Transformers in 3D Medical Imaging
• Learning from Mistakes based on Class Weighting with Application to Neural Architecture Search
• Enhanced MRI Reconstruction Network using Neural Architecture Search
• Efficient Data-specific Model Search for Collaborative Filtering
• Jets and differential linear logic
• Out-of-the-box channel pruned networks
• Revisiting Neural Architecture Search
• Efficient Transfer Learning via Joint Adaptation of Network Architecture and Weight
• Efficient Modelling Across Time of Human Actions and Interactions
• Max and Coincidence Neurons in Neural Networks
• Deformable Stacked Structure for Named Entity Recognition
• Neural Architecture Search for Image Super-Resolution Using Densely Constructed Search Space: DeCoNAS
• "You might also like this model": Data Driven Approach for Recommending Deep Learning Models for Unknown Image Datasets
• Making Differentiable Architecture Search less local
• CP-NAS: Child-Parent Neural Architecture Search for Binary Neural Networks
• Memory-Efficient Hierarchical Neural Architecture Search for Image Restoration
• Neural Architecture Search with an Efficient Multiobjective Evolutionary Framework
• Improving Neural Language Models by Segmenting, Attending, and Predicting the Future
• Grow-Push-Prune: aligning deep discriminants for effective structural network compression
• Effective Model Compression via Stage-wise Pruning
• Channel Planting for Deep Neural Networks using Knowledge Distillation
• GPNAS: A Neural Network Architecture Search Framework Based on Graphical Predictor
• Neural Architecture Search based on Cartesian Genetic Programming Coding Method
• Understanding the wiring evolution in differentiable neural architecture search
• Small-Group Learning, with Application to Neural Architecture Search