You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization

Abstract

Recently Neural Architecture Search (NAS) has aroused great interest in both academia and industry, however it remains challenging because of its huge and non-continuous search space. Instead of applying evolutionary algorithm or reinforcement learning as previous works, this paper proposes a Direct Sparse Optimization NAS (DSO-NAS) method. In DSO-NAS, we provide a novel model pruning view to NAS problem. In specific, we start from a completely connected block, and then introduce scaling factors to scale the information flow between operations. Next, we impose sparse regularizations to prune useless connections in the architecture. Lastly, we derive an efficient and theoretically sound optimization method to solve it. Our method enjoys both advantages of differentiability and efficiency, therefore can be directly applied to large datasets like ImageNet. Particularly, On CIFAR-10 dataset, DSO-NAS achieves an average test error 2.84\%, while on the ImageNet dataset DSO-NAS achieves 25.4\% test error under 600M FLOPs with 8 GPUs in 18 hours.

arXiv commentICLR2019 Submission

References in corpus (7)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• FractalNet: Ultra-Deep Neural Networks without Residuals
• Learning Structured Sparsity in Deep Neural Networks
• Progressive Neural Architecture Search

Cited by in corpus (18)

• AutoML: A Survey of the State-of-the-Art
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• ASAP: Architecture Search, Anneal and Prune
• A Comprehensive Survey on Hardware-Aware Neural Architecture Search
• ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Coding
• Deeper Insights into Weight Sharing in Neural Architecture Search
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• XNAS: Neural Architecture Search with Expert Advice
• V-NAS: Neural Architecture Search for Volumetric Medical Image Segmentation
• Efficient Decoupled Neural Architecture Search by Structure and Operation Sampling
• Evolving Search Space for Neural Architecture Search
• Neural Architecture Refinement: A Practical Way for Avoiding Overfitting in NAS
• RAPDARTS: Resource-Aware Progressive Differentiable Architecture Search
• MemNet: Memory-Efficiency Guided Neural Architecture Search with Augment-Trim learning
• How Does Supernet Help in Neural Architecture Search?
• NASIB: Neural Architecture Search withIn Budget
• Neural networks adapting to datasets: learning network size and topology
• Enhanced Gradient for Differentiable Architecture Search