L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks

Abstract

Batch Normalization (BN) has been proven to be quite effective at accelerating and improving the training of deep neural networks (DNNs). However, BN brings additional computation, consumes more memory and generally slows down the training process by a large margin, which aggravates the training effort. Furthermore, the nonlinear square and root operations in BN also impede the low bit-width quantization techniques, which draws much attention in deep learning hardware community. In this work, we propose an L1-norm BN (L1BN) with only linear operations in both the forward and the backward propagations during training. L1BN is shown to be approximately equivalent to the original L2-norm BN (L2BN) by multiplying a scaling factor. Experiments on various convolutional neural networks (CNNs) and generative adversarial networks (GANs) reveal that L1BN maintains almost the same accuracies and convergence rates compared to L2BN but with higher computational efficiency. On FPGA platform, the proposed signum and absolute operations in L1BN can achieve 1.5$\times$ speedup and save 50\% power consumption, compared with the original costly square and root operations, respectively. This hardware-friendly normalization method not only surpasses L2BN in speed, but also simplify the hardware design of ASIC accelerators with higher energy efficiency. Last but not the least, L1BN promises a fully quantized training of DNNs, which is crucial to future adaptive terminal devices.

arXiv comment8 pages, 4 figures

References in corpus (20)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• TensorFlow: A system for large-scale machine learning
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Deeply-Supervised Nets
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Binarized Neural Networks
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
• Layer Normalization
• Xception: Deep Learning with Depthwise Separable Convolutions
• Training and Inference with Integers in Deep Neural Networks
• Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
• Bayesian Uncertainty Estimation for Batch Normalized Deep Networks
• Effective Quantization Methods for Recurrent Neural Networks

Cited by in corpus (14)

• A survey on GANs for computer vision: Recent research, analysis and taxonomy
• Artificial-Intelligence-Based Design for Circuit Parameters of Power Converters
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey
• Vision Transformers on the Edge: A Comprehensive Survey of Model Compression and Acceleration Strategies
• A Comprehensive and Modularized Statistical Framework for Gradient Norm Equality in Deep Neural Networks
• ES-ImageNet: A Million Event-Stream Classification Dataset for Spiking Neural Networks
• IGUANe: a 3D generalizable CycleGAN for multicenter harmonization of brain MR images
• A Robust Initialization of Residual Blocks for Effective ResNet Training without Batch Normalization
• Activated Gradients for Deep Neural Networks
• Layer-wise Conditioning Analysis in Exploring the Learning Dynamics of DNNs
• Enabling Binary Neural Network Training on the Edge
• Batch Normalization Sampling
• Training and Inference for Integer-Based Semantic Segmentation Network