Improving One-shot NAS by Suppressing the Posterior Fading

Abstract

There is a growing interest in automated neural architecture search (NAS). To improve the efficiency of NAS, previous approaches adopt weight sharing method to force all models share the same set of weights. However, it has been observed that a model performing better with shared weights does not necessarily perform better when trained alone. In this paper, we analyse existing weight sharing one-shot NAS approaches from a Bayesian point of view and identify the posterior fading problem, which compromises the effectiveness of shared weights. To alleviate this problem, we present a practical approach to guide the parameter posterior towards its true distribution. Moreover, a hard latency constraint is introduced during the search so that the desired latency can be achieved. The resulted method, namely Posterior Convergent NAS (PC-NAS), achieves state-of-the-art performance under standard GPU latency constraint on ImageNet. In our small search space, our model PC-NAS-S attains 76.8 % top-1 accuracy, 2.1% higher than MobileNetV2 (1.4x) with the same latency. When adopted to the large search space, PC-NAS-L achieves 78.1 % top-1 accuracy within 11ms. The discovered architecture also transfers well to other computer vision applications such as object detection and person re-identification.

References in corpus (13)

• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Neural Architecture Search with Reinforcement Learning
• Efficient Neural Architecture Search via Parameter Sharing
• Large-Scale Evolution of Image Classifiers
• Designing Neural Network Architectures using Reinforcement Learning
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
• ASAP: Architecture Search, Anneal and Prune
• AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search
• Understanding Neural Architecture Search Techniques
• XNAS: Neural Architecture Search with Expert Advice
• AM-LFS: AutoML for Loss Function Search
• Online Hyper-parameter Learning for Auto-Augmentation Strategy

Cited by in corpus (6)

• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• Blockwisely Supervised Neural Architecture Search with Knowledge Distillation
• How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS
• Angle-based Search Space Shrinking for Neural Architecture Search
• Powering One-shot Topological NAS with Stabilized Share-parameter Proxy
• BS-NAS: Broadening-and-Shrinking One-Shot NAS with Searchable Numbers of Channels