Single-Path Mobile AutoML: Efficient ConvNet Design and NAS Hyperparameter Optimization
arXiv:1907.00959 · doi:10.1109/JSTSP.2020.2971421
Abstract
Can we reduce the search cost of Neural Architecture Search (NAS) from days down to only few hours? NAS methods automate the design of Convolutional Networks (ConvNets) under hardware constraints and they have emerged as key components of AutoML frameworks. However, the NAS problem remains challenging due to the combinatorially large design space and the significant search time (at least 200 GPU-hours). In this work, we alleviate the NAS search cost down to less than 3 hours, while achieving state-of-the-art image classification results under mobile latency constraints. We propose a novel differentiable NAS formulation, namely Single-Path NAS, that uses one single-path over-parameterized ConvNet to encode all architectural decisions based on shared convolutional kernel parameters, hence drastically decreasing the search overhead. Single-Path NAS achieves state-of-the-art top-1 ImageNet accuracy (75.62%), hence outperforming existing mobile NAS methods in similar latency settings (~80ms). In particular, we enhance the accuracy-runtime trade-off in differentiable NAS by treating the Squeeze-and-Excitation path as a fully searchable operation with our novel single-path encoding. Our method has an overall cost of only 8 epochs (24 TPU-hours), which is up to 5,000x faster compared to prior work. Moreover, we study how different NAS formulation choices affect the performance of the designed ConvNets. Furthermore, we exploit the efficiency of our method to answer an interesting question: instead of empirically tuning the hyperparameters of the NAS solver (as in prior work), can we automatically find the hyperparameter values that yield the desired accuracy-runtime trade-off? We open-source our entire codebase at: https://github.com/dstamoulis/single-path-nas.
arXiv commentDetailed extension (journal) of the Single-Path NAS ECMLPKDD'19 paper (arXiv:1904.02877)
References in corpus (13)
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Neural Architecture Search with Reinforcement Learning
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- DARTS: Differentiable Architecture Search
- PACT: Parameterized Clipping Activation for Quantized Neural Networks
- Random Search and Reproducibility for Neural Architecture Search
- Single Path One-Shot Neural Architecture Search with Uniform Sampling
- MONAS: Multi-Objective Neural Architecture Search using Reinforcement Learning
- Multi-fidelity Bayesian Optimisation with Continuous Approximations
- NeuralPower: Predict and Deploy Energy-Efficient Convolutional Neural Networks
- Layer-compensated Pruning for Resource-constrained Convolutional Neural Networks
- sharpDARTS: Faster and More Accurate Differentiable Architecture Search
- One-Shot Neural Architecture Search via Compressive Sensing