Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Training
arXiv:2106.03640
Abstract
Much recent research has been dedicated to improving the efficiency of training and inference for image classification. This effort has commonly focused on explicitly improving theoretical efficiency, often measured as ImageNet validation accuracy per FLOP. These theoretical savings have, however, proven challenging to achieve in practice, particularly on high-performance training accelerators. In this work, we focus on improving the practical efficiency of the state-of-the-art EfficientNet models on a new class of accelerator, the Graphcore IPU. We do this by extending this family of models in the following ways: (i) generalising depthwise convolutions to group convolutions; (ii) adding proxy-normalized activations to match batch normalization performance with batch-independent statistics; (iii) reducing compute by lowering the training resolution and inexpensively fine-tuning at higher resolution. We find that these three methods improve the practical efficiency for both training and inference. Code available at https://github.com/graphcore/graphcore-research/tree/main/Making_EfficientNet_More_Efficient .
References in corpus (33)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
- mixup: Beyond Empirical Risk Minimization
- Instance Normalization: The Missing Ingredient for Fast Stylization
- EfficientNetV2: Smaller Models and Faster Training
- ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
- Fully Convolutional Networks for Semantic Segmentation
- AutoAugment: Learning Augmentation Policies from Data
- CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features
- Training Deep Nets with Sublinear Memory Cost
- Aggregated Residual Transformations for Deep Neural Networks
- Xception: Deep Learning with Depthwise Separable Convolutions
- Revisiting Small Batch Training for Deep Neural Networks
- Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups
- High-Performance Large-Scale Image Recognition Without Normalization
- Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
- Revisiting ResNets: Improved Training and Scaling Strategies
- Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
- Micro-Batch Training with Batch-Channel Normalization and Weight Standardization
- Image Classification at Supercomputer Scale
- PipeDream: Fast and Efficient Pipeline Parallel DNN Training
- Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks
- Fixing the train-test resolution discrepancy: FixEfficientNet
- Evolving Normalization-Activation Layers
- Memory-Efficient Backpropagation Through Time
- Mesh-TensorFlow: Deep Learning for Supercomputers
- Compounding the Performance Improvements of Assembled Techniques in a Convolutional Neural Network
- Online Normalization for Training Neural Networks
- Fixing the train-test resolution discrepancy
- Characterizing signal propagation to close the performance gap in unnormalized ResNets
- Neural Architecture Design for GPU-Efficient Networks
- Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence