Single Path One-Shot Neural Architecture Search with Uniform Sampling

Abstract

We revisit the one-shot Neural Architecture Search (NAS) paradigm and analyze its advantages over existing NAS approaches. Existing one-shot method, however, is hard to train and not yet effective on large scale datasets like ImageNet. This work propose a Single Path One-Shot model to address the challenge in the training. Our central idea is to construct a simplified supernet, where all architectures are single paths so that weight co-adaption problem is alleviated. Training is performed by uniform path sampling. All architectures (and their weights) are trained fully and equally. Comprehensive experiments verify that our approach is flexible and effective. It is easy to train and fast to search. It effortlessly supports complex search spaces (e.g., building blocks, channel, mixed-precision quantization) and different search constraints (e.g., FLOPs, latency). It is thus convenient to use for various needs. It achieves start-of-the-art performance on the large dataset ImageNet.

arXiv commentECCV 2020

References in corpus (22)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Deep Residual Learning for Image Recognition
• Neural Architecture Search with Reinforcement Learning
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Efficient Neural Architecture Search via Parameter Sharing
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• Channel Pruning for Accelerating Very Deep Neural Networks
• SNAS: Stochastic Neural Architecture Search
• Random Search and Reproducibility for Neural Architecture Search
• Hierarchical Representations for Efficient Architecture Search
• Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
• Evaluating the Search Phase of Neural Architecture Search
• NAS evaluation is frustratingly hard
• FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
• NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
• NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications
• You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization
• BlockQNN: Efficient Block-wise Neural Network Architecture Generation
• ChamNet: Towards Efficient Network Design through Platform-Aware Model Adaptation
• Efficient Neural Architecture Search via Proximal Iterations

Cited by in corpus (127)

• Transformers in Vision: A Survey
• A Survey on Evolutionary Neural Architecture Search
• MCUNet: Tiny Deep Learning on IoT Devices
• HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
• Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• DetNAS: Backbone Search for Object Detection
• Evaluating the Search Phase of Neural Architecture Search
• Once-for-All: Train One Network and Specialize it for Efficient Deployment
• SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers
• Dynamic Convolution: Attention over Convolution Kernels
• Sample and Computation Redistribution for Efficient Face Detection
• Auto-GNN: Neural Architecture Search of Graph Neural Networks
• A Comprehensive Survey on Hardware-Aware Neural Architecture Search
• Compacting Deep Neural Networks for Internet of Things: Methods and Applications
• Evolving Normalization-Activation Layers
• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
• Semi-Supervised Neural Architecture Search
• Stabilizing DARTS with Amended Gradient Estimation on Architectural Parameters
• AtomNAS: Fine-Grained End-to-End Neural Architecture Search
• BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models
• ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Coding
• Deeper Insights into Weight Sharing in Neural Architecture Search
• Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?
• Sample-Efficient Neural Architecture Search by Learning Action Space
• TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learning
• Computation Reallocation for Object Detection
• GOLD-NAS: Gradual, One-Level, Differentiable
• FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Blockwisely Supervised Neural Architecture Search with Knowledge Distillation
• DSNAS: Direct Neural Architecture Search without Parameter Retraining
• Latency-Aware Differentiable Neural Architecture Search
• CARS: Continuous Evolution for Efficient Neural Architecture Search
• How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS
• Joint Multi-Dimension Pruning via Numerical Gradient Update
• NAS-FCOS: Fast Neural Architecture Search for Object Detection
• APQ: Joint Search for Network Architecture, Pruning and Quantization Policy
• FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining
• STEERAGE: Synthesis of Neural Networks Using Architecture Search and Grow-and-Prune Methods
• C2FNAS: Coarse-to-Fine Neural Architecture Search for 3D Medical Image Segmentation
• Auto-Panoptic: Cooperative Multi-Component Architecture Search for Panoptic Segmentation
• Learned Low Precision Graph Neural Networks
• Real-time Federated Evolutionary Neural Architecture Search
• Angle-based Search Space Shrinking for Neural Architecture Search
• MixPath: A Unified Approach for One-shot Neural Architecture Search
• BatchQuant: Quantized-for-all Architecture Search with Robust Quantizer
• Rethinking Bottleneck Structure for Efficient Mobile Network Design
• To Share or Not To Share: A Comprehensive Appraisal of Weight-Sharing
• Stronger NAS with Weaker Predictors
• Learning Dynamic Routing for Semantic Segmentation
• BETANAS: BalancEd TrAining and selective drop for Neural Architecture Search
• Efficient Residual Dense Block Search for Image Super-Resolution
• Single-Path NAS: Device-Aware Efficient ConvNet Design
• GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernet
• Rethinking Differentiable Search for Mixed-Precision Neural Networks
• A Semi-Supervised Assessor of Neural Architectures
• Cyclic Differentiable Architecture Search
• A Variational-Sequential Graph Autoencoder for Neural Architecture Performance Prediction
• From Federated Learning to Federated Neural Architecture Search: A Survey
• Efficient Deep Neural Networks
• EcoNAS: Finding Proxies for Economical Neural Architecture Search
• Hierarchical Neural Architecture Search via Operator Clustering
• Improving One-shot NAS by Suppressing the Posterior Fading
• Effective Training of Convolutional Neural Networks with Low-bitwidth Weights and Activations
• MTL-NAS: Task-Agnostic Neural Architecture Search towards General-Purpose Multi-Task Learning
• Few-shot Neural Architecture Search
• Learning Versatile Convolution Filters for Efficient Visual Recognition
• Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search
• DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture
• PV-NAS: Practical Neural Architecture Search for Video Recognition
• S3NAS: Fast NPU-aware Neural Architecture Search Methodology
• Powering One-shot Topological NAS with Stabilized Share-parameter Proxy
• Domain-Aware Dynamic Networks
• Fitting the Search Space of Weight-sharing NAS with Graph Convolutional Networks
• HM-NAS: Efficient Neural Architecture Search via Hierarchical Masking
• HourNAS: Extremely Fast Neural Architecture Search Through an Hourglass Lens
• Binarizing MobileNet via Evolution-based Searching
• FNAS: Uncertainty-Aware Fast Neural Architecture Search
• Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-based Approach
• Integrating Large Circular Kernels into CNNs through Neural Architecture Search
• ZARTS: On Zero-order Optimization for Neural Architecture Search
• Towards Learning of Filter-Level Heterogeneous Compression of Convolutional Neural Networks
• DOTS: Decoupling Operation and Topology in Differentiable Architecture Search
• MoGA: Searching Beyond MobileNetV3
• FNA++: Fast Network Adaptation via Parameter Remapping and Architecture Search
• BS-NAS: Broadening-and-Shrinking One-Shot NAS with Searchable Numbers of Channels
• MetaMix: Meta-state Precision Searcher for Mixed-precision Activation Quantization
• Single-Path Mobile AutoML: Efficient ConvNet Design and NAS Hyperparameter Optimization
• PONAS: Progressive One-shot Neural Architecture Search for Very Efficient Deployment
• Neural Architecture Search for Lightweight Non-Local Networks
• Scaling Up Quantization-Aware Neural Architecture Search for Efficient Deep Learning on the Edge
• SCARLET-NAS: Bridging the Gap between Stability and Scalability in Weight-sharing Neural Architecture Search
• RAPDARTS: Resource-Aware Progressive Differentiable Architecture Search
• Fast Hardware-Aware Neural Architecture Search
• CompOFA: Compound Once-For-All Networks for Faster Multi-Platform Deployment
• How Does Supernet Help in Neural Architecture Search?
• Tiny Adversarial Mulit-Objective Oneshot Neural Architecture Search
• Spatiotemporal Fusion in 3D CNNs: A Probabilistic View
• DA-NAS: Data Adapted Pruning for Efficient Neural Architecture Search
• Single-level Optimization For Differential Architecture Search
• Neural Architecture Search with Random Labels
• Memory-Efficient Differentiable Transformer Architecture Search
• Neural Inheritance Relation Guided One-Shot Layer Assignment Search
• AutoADR: Automatic Model Design for Ad Relevance
• Rapid Elastic Architecture Search under Specialized Classes and Resource Constraints
• Disentangled Neural Architecture Search
• BaLeNAS: Differentiable Architecture Search via the Bayesian Learning Rule
• NAAS: Neural Accelerator Architecture Search
• WAS-VTON: Warping Architecture Search for Virtual Try-on Network
• AutoPruning for Deep Neural Network with Dynamic Channel Masking
• Dynamic Slimmable Network
• Efficient Novelty-Driven Neural Architecture Search
• Evolutionary Algorithm Enhanced Neural Architecture Search for Text-Independent Speaker Verification
• Differentiable Network Adaption with Elastic Search Space
• DICE: Deep Significance Clustering for Outcome-Aware Stratification
• Network Adjustment: Channel Search Guided by FLOPs Utilization Ratio
• NAS-FCOS: Efficient Search for Object Detection Architectures
• Mutually-aware Sub-Graphs Differentiable Architecture Search
• LV-BERT: Exploiting Layer Variety for BERT
• Network representation learning: A macro and micro view
• T-AutoML: Automated Machine Learning for Lesion Segmentation using Transformers in 3D Medical Imaging
• Connection Sensitivity Matters for Training-free DARTS: From Architecture-Level Scoring to Operation-Level Sensitivity Analysis
• Memory-Efficient Hierarchical Neural Architecture Search for Image Restoration
• Effective Model Compression via Stage-wise Pruning
• Understanding the wiring evolution in differentiable neural architecture search
• Joint Learning of Neural Transfer and Architecture Adaptation for Image Recognition