ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Coding

Abstract

Neural architecture search (NAS) aims to produce the optimal sparse solution from a high-dimensional space spanned by all candidate connections. Current gradient-based NAS methods commonly ignore the constraint of sparsity in the search phase, but project the optimized solution onto a sparse one by post-processing. As a result, the dense super-net for search is inefficient to train and has a gap with the projected architecture for evaluation. In this paper, we formulate neural architecture search as a sparse coding problem. We perform the differentiable search on a compressed lower-dimensional space that has the same validation loss as the original sparse solution space, and recover an architecture by solving the sparse coding problem. The differentiable search and architecture recovery are optimized in an alternate manner. By doing so, our network for search at each update satisfies the sparsity constraint and is efficient to train. In order to also eliminate the depth and width gap between the network in search and the target-net in evaluation, we further propose a method to search and evaluate in one stage under the target-net settings. When training finishes, architecture variables are absorbed into network weights. Thus we get the searched architecture and optimized parameters in a single run. In experiments, our two-stage method on CIFAR-10 requires only 0.05 GPU-day for search. Our one-stage method produces state-of-the-art performances on both CIFAR-10 and ImageNet at the cost of only evaluation time.

arXiv commentNeurIPS 2020

References in corpus (18)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• Categorical Reparameterization with Gumbel-Softmax
• DARTS: Differentiable Architecture Search
• A survey of sparse representation: algorithms and applications
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Large-Scale Evolution of Image Classifiers
• Neural Architecture Optimization
• Searching for Efficient Multi-Scale Architectures for Dense Image Prediction
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• Path-Level Network Transformation for Efficient Architecture Search
• NAS evaluation is frustratingly hard
• BayesNAS: A Bayesian Approach for Neural Architecture Search
• AtomNAS: Fine-Grained End-to-End Neural Architecture Search
• You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization
• One-Shot Neural Architecture Search via Compressive Sensing
• Optimization Algorithm Inspired Deep Neural Network Structure Design

Cited by in corpus (9)

• Quantum circuit architecture search for variational quantum algorithms
• Learning Deep Morphological Networks with Neural Architecture Search
• Locally Free Weight Sharing for Network Width Search
• DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture
• RNAS-CL: Robust Neural Architecture Search by Cross-Layer Knowledge Distillation
• Prioritized Architecture Sampling with Monto-Carlo Tree Search
• BCNet: Searching for Network Width with Bilaterally Coupled Network
• Memory-Efficient Differentiable Transformer Architecture Search
• GreedyNASv2: Greedier Search with a Greedy Path Filter