Restructuring Batch Normalization to Accelerate CNN Training
arXiv:1807.01702
Abstract
Batch Normalization (BN) has become a core design block of modern Convolutional Neural Networks (CNNs). A typical modern CNN has a large number of BN layers in its lean and deep architecture. BN requires mean and variance calculations over each mini-batch during training. Therefore, the existing memory access reduction techniques, such as fusing multiple CONV layers, are not effective for accelerating BN due to their inability to optimize mini-batch related calculations during training. To address this increasingly important problem, we propose to restructure BN layers by first splitting a BN layer into two sub-layers (fission) and then combining the first sub-layer with its preceding CONV layer and the second sub-layer with the following activation and CONV layers (fusion). The proposed solution can significantly reduce main-memory accesses while training the latest CNN models, and the experiments on a chip multiprocessor show that the proposed BN restructuring can improve the performance of DenseNet-121 by 25.7%.
arXiv comment13 pages, 8 figures, to appear in SysML 2019, added ResNet-50 results
References in corpus (14)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- TensorFlow: A system for large-scale machine learning
- Practical Bayesian Optimization of Machine Learning Algorithms
- Caffe: Convolutional Architecture for Fast Feature Embedding
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
- cuDNN: Efficient Primitives for Deep Learning
- An Analysis of Deep Neural Network Models for Practical Applications
- Aggregated Residual Transformations for Deep Neural Networks
- Reducing Overfitting in Deep Networks by Decorrelating Representations
- A Systematic Approach to Blocking Convolutional Neural Networks
- Partitioning Compute Units in CNN Acceleration for Statistical Memory Traffic Shaping