BN-NAS: Neural Architecture Search with Batch Normalization

Abstract

We present BN-NAS, neural architecture search with Batch Normalization (BN-NAS), to accelerate neural architecture search (NAS). BN-NAS can significantly reduce the time required by model training and evaluation in NAS. Specifically, for fast evaluation, we propose a BN-based indicator for predicting subnet performance at a very early training stage. The BN-based indicator further facilitates us to improve the training efficiency by only training the BN parameters during the supernet training. This is based on our observation that training the whole supernet is not necessary while training only BN parameters accelerates network convergence for network architecture search. Extensive experiments show that our method can significantly shorten the time of training supernet by more than 10 times and shorten the time of evaluating subnets by more than 600,000 times without losing accuracy.

ICCV 2021

References in corpus (6)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Learning Efficient Convolutional Networks through Network Slimming
• Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation
• Operation-Aware Soft Channel Pruning using Differentiable Masks
• Computation Reallocation for Object Detection
• FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions