Neural Architecture Refinement: A Practical Way for Avoiding Overfitting in NAS

Abstract

Neural architecture search (NAS) is proposed to automate the architecture design process and attracts overwhelming interest from both academia and industry. However, it is confronted with overfitting issue due to the high-dimensional search space composed by operator selection and skip connection of each layer. This paper explores the architecture overfitting issue in depth based on the reinforcement learning-based NAS framework. We show that the policy gradient method has deep correlations with the cross entropy minimization. Based on this correlation, we further demonstrate that, though the reward of NAS is sparse, the policy gradient method implicitly assign the reward to all operations and skip connections based on the sampling frequency. However, due to the inaccurate reward estimation, curse of dimensionality problem and the hierachical structure of neural networks, reward charateristics for operators and skip connections have intrinsic differences, the assigned rewards for the skip connections are extremely noisy and inaccurate. To alleviate this problem, we propose a neural architecture refinement approach that working with an initial state-of-the-art network structure and only refining its operators. Extensive experiments have demonstrated that the proposed method can achieve fascinated results, including classification, face recognition etc.

arXiv comment9 pages, 1 figures, 5 tables

References in corpus (14)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• WaveNet: A Generative Model for Raw Audio
• Convolutional Sequence to Sequence Learning
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Efficient Neural Architecture Search via Parameter Sharing
• Searching for Efficient Multi-Scale Architectures for Dense Image Prediction
• SNAS: Stochastic Neural Architecture Search
• Hierarchical Representations for Efficient Architecture Search
• Neural Optimizer Search with Reinforcement Learning
• Path-Level Network Transformation for Efficient Architecture Search
• Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation
• Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation
• FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
• You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization

Cited by in corpus (3)

• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Efficient Neural Architecture Search with Performance Prediction
• GPNAS: A Neural Network Architecture Search Framework Based on Graphical Predictor