BN-NAS: Neural Architecture Search with Batch Normalization
arXiv:2108.07375
Abstract
We present BN-NAS, neural architecture search with Batch Normalization (BN-NAS), to accelerate neural architecture search (NAS). BN-NAS can significantly reduce the time required by model training and evaluation in NAS. Specifically, for fast evaluation, we propose a BN-based indicator for predicting subnet performance at a very early training stage. The BN-based indicator further facilitates us to improve the training efficiency by only training the BN parameters during the supernet training. This is based on our observation that training the whole supernet is not necessary while training only BN parameters accelerates network convergence for network architecture search. Extensive experiments show that our method can significantly shorten the time of training supernet by more than 10 times and shorten the time of evaluating subnets by more than 600,000 times without losing accuracy.
arXiv commentICCV 2021
References in corpus (13)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- MobileNetV2: Inverted Residuals and Linear Bottlenecks
- DARTS: Differentiable Architecture Search
- Efficient Neural Architecture Search via Parameter Sharing
- Searching for Efficient Multi-Scale Architectures for Dense Image Prediction
- Learning Efficient Convolutional Networks through Network Slimming
- Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation
- Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs
- Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks
- Operation-Aware Soft Channel Pruning using Differentiable Masks
- Computation Reallocation for Object Detection
- FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions
- Angle-based Search Space Shrinking for Neural Architecture Search