Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Training
arXiv:2106.03640
Abstract
Much recent research has been dedicated to improving the efficiency of training and inference for image classification. This effort has commonly focused on explicitly improving theoretical efficiency, often measured as ImageNet validation accuracy per FLOP. These theoretical savings have, however, proven challenging to achieve in practice, particularly on high-performance training accelerators. In this work, we focus on improving the practical efficiency of the state-of-the-art EfficientNet models on a new class of accelerator, the Graphcore IPU. We do this by extending this family of models in the following ways: (i) generalising depthwise convolutions to group convolutions; (ii) adding proxy-normalized activations to match batch normalization performance with batch-independent statistics; (iii) reducing compute by lowering the training resolution and inexpensively fine-tuning at higher resolution. We find that these three methods improve the practical efficiency for both training and inference. Code available at https://github.com/graphcore/graphcore-research/tree/main/Making_EfficientNet_More_Efficient .
References in corpus (7)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
- High-Performance Large-Scale Image Recognition Without Normalization
- Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
- Revisiting ResNets: Improved Training and Scaling Strategies
- Characterizing signal propagation to close the performance gap in unnormalized ResNets
- Neural Architecture Design for GPU-Efficient Networks