Training BatchNorm Only in Neural Architecture Search and Beyond

Abstract

This work investigates the usage of batch normalization in neural architecture search (NAS). Specifically, Frankle et al. find that training BatchNorm only can achieve nontrivial performance. Furthermore, Chen et al. claim that training BatchNorm only can speed up the training of the one-shot NAS supernet over ten times. Critically, there is no effort to understand 1) why training BatchNorm only can find the perform-well architectures with the reduced supernet-training time, and 2) what is the difference between the train-BN-only supernet and the standard-train supernet. We begin by showing that the train-BN-only networks converge to the neural tangent kernel regime, obtain the same training dynamics as train all parameters theoretically. Our proof supports the claim to train BatchNorm only on supernet with less training time. Then, we empirically disclose that train-BN-only supernet provides an advantage on convolutions over other operators, cause unfair competition between architectures. This is due to only the convolution operator being attached with BatchNorm. Through experiments, we show that such unfairness makes the search algorithm prone to select models with convolutions. To solve this issue, we introduce fairness in the search space by placing a BatchNorm layer on every operator. However, we observe that the performance predictor in Chen et al. is inapplicable on the new search space. To this end, we propose a novel composite performance indicator to evaluate networks from three perspectives: expressivity, trainability, and uncertainty, derived from the theoretical property of BatchNorm. We demonstrate the effectiveness of our approach on multiple NAS-benchmarks (NAS-Bench101, NAS-Bench-201) and search spaces (DARTS search space and MobileNet search space).

arXiv comment11 pages Technical report

References in corpus (30)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distilling the Knowledge in a Neural Network
• Knowledge Distillation: A Survey
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• DARTS: Differentiable Architecture Search
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• On the Number of Linear Regions of Deep Neural Networks
• Neural Architecture Optimization
• Scalable Bayesian Optimization Using Deep Neural Networks
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Exponential expressivity in deep neural networks through transient chaos
• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• Model compression via distillation and quantization
• One-Shot Neural Architecture Search via Self-Evaluated Template Network
• Bayesian Uncertainty Estimation for Batch Normalized Deep Networks
• Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs
• Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS
• AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search
• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
• Semi-Supervised Neural Architecture Search
• Rethinking "Batch" in BatchNorm
• How Powerful are Performance Predictors in Neural Architecture Search?
• Evolutionary-Neural Hybrid Agents for Architecture Search
• Partial transfusion: on the expressive influence of trainable batch norm parameters for transfer learning
• BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search
• Batch Normalization Orthogonalizes Representations in Deep Random Networks
• Bag of Tricks for Neural Architecture Search
• BN-NAS: Neural Architecture Search with Batch Normalization