ASAP: Architecture Search, Anneal and Prune

Abstract

Automatic methods for Neural Architecture Search (NAS) have been shown to produce state-of-the-art network models. Yet, their main drawback is the computational complexity of the search process. As some primal methods optimized over a discrete search space, thousands of days of GPU were required for convergence. A recent approach is based on constructing a differentiable search space that enables gradient-based optimization, which reduces the search time to a few days. While successful, it still includes some noncontinuous steps, e.g., the pruning of many weak connections at once. In this paper, we propose a differentiable search space that allows the annealing of architecture weights, while gradually pruning inferior operations. In this way, the search converges to a single output network in a continuous manner. Experiments on several vision datasets demonstrate the effectiveness of our method with respect to the search cost and accuracy of the achieved model. Specifically, with $0.2$ GPU search days we achieve an error rate of $1.68\%$ on CIFAR-10.

References in corpus (24)

• Adam: A Method for Stochastic Optimization
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• Neural Architecture Search: A Survey
• SGDR: Stochastic Gradient Descent with Warm Restarts
• DARTS: Differentiable Architecture Search
• Random Erasing Data Augmentation
• AutoAugment: Learning Augmentation Policies from Data
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• FractalNet: Ultra-Deep Neural Networks without Residuals
• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• Large-Scale Evolution of Image Classifiers
• Neural Architecture Optimization
• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• Compression-aware Training of Deep Networks
• Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation
• Probabilistic Neural Architecture Search
• The Freiburg Groceries Dataset
• You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization
• sharpDARTS: Faster and More Accurate Differentiable Architecture Search
• Full deep neural network training on a pruned weight budget
• Prune and Replace NAS

Cited by in corpus (21)

• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• Differentiable Quantum Architecture Search
• AutoML-Zero: Evolving Machine Learning Algorithms From Scratch
• Stabilizing DARTS with Amended Gradient Estimation on Architectural Parameters
• XNAS: Neural Architecture Search with Expert Advice
• Angle-based Search Space Shrinking for Neural Architecture Search
• BETANAS: BalancEd TrAining and selective drop for Neural Architecture Search
• Self-supervised Neural Architecture Search
• Cyclic Differentiable Architecture Search
• Efficient Neural Architecture Search via Proximal Iterations
• Improving One-shot NAS by Suppressing the Posterior Fading
• Prune and Replace NAS
• Neural Epitome Search for Architecture-Agnostic Network Compression
• Graph-guided Architecture Search for Real-time Semantic Segmentation
• Towards Learning of Filter-Level Heterogeneous Compression of Convolutional Neural Networks
• Geometry-Aware Gradient Algorithms for Neural Architecture Search
• Multi-Objective Neural Architecture Search Based on Diverse Structures and Adaptive Recommendation
• Fisher Task Distance and Its Application in Neural Architecture Search
• Hyperparameter Optimization in Neural Networks via Structured Sparse Recovery
• iDARTS: Improving DARTS by Node Normalization and Decorrelation Discretization
• FIVES: Feature Interaction Via Edge Search for Large-Scale Tabular Data