Latency-Aware Differentiable Neural Architecture Search

Abstract

Differentiable neural architecture search methods became popular in recent years, mainly due to their low search costs and flexibility in designing the search space. However, these methods suffer the difficulty in optimizing network, so that the searched network is often unfriendly to hardware. This paper deals with this problem by adding a differentiable latency loss term into optimization, so that the search process can tradeoff between accuracy and latency with a balancing coefficient. The core of latency prediction is to encode each network architecture and feed it into a multi-layer regressor, with the training data which can be easily collected from randomly sampling a number of architectures and evaluating them on the hardware. We evaluate our approach on NVIDIA Tesla-P100 GPUs. With 100K sampled architectures (requiring a few hours), the latency prediction module arrives at a relative error of lower than 10%. Equipped with this module, the search method can reduce the latency by 20% meanwhile preserving the accuracy. Our approach also enjoys the ability of being transplanted to a wide range of hardware platforms with very few efforts, or being used to optimizing other non-differentiable factors such as power consumption.

19 pages, 9 figures

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Neural Architecture Search: A Survey
• Large-Scale Evolution of Image Classifiers
• Neural Architecture Optimization
• Designing Neural Network Architectures using Reinforcement Learning
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• SNAS: Stochastic Neural Architecture Search
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• A Survey on Neural Architecture Search
• Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation

Cited by in corpus (8)

• A Comprehensive Survey on Hardware-Aware Neural Architecture Search
• GOLD-NAS: Gradual, One-Level, Differentiable
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Weight-dependent Gates for Network Pruning
• MS-DARTS: Mean-Shift Based Differentiable Architecture Search
• Rapid Model Architecture Adaption for Meta-Learning
• RHNAS: Realizable Hardware and Neural Architecture Search
• Latency-Controlled Neural Architecture Search for Streaming Speech Recognition