Scalable Methods for 8-bit Training of Neural Networks

Abstract

Quantized Neural Networks (QNNs) are often used to improve network efficiency during the inference phase, i.e. after the network has been trained. Extensive research in the field suggests many different quantization schemes. Still, the number of bits required, as well as the best quantization scheme, are yet unknown. Our theoretical analysis suggests that most of the training process is robust to substantial precision reduction, and points to only a few specific operations that require higher precision. Armed with this knowledge, we quantize the model parameters, activations and layer gradients to 8-bit, leaving at a higher precision only the final step in the computation of the weight gradients. Additionally, as QNNs require batch-normalization to be trained at high precision, we introduce Range Batch-Normalization (BN) which has significantly higher tolerance to quantization noise and improved computational complexity. Our simulations show that Range BN is equivalent to the traditional batch norm if a precise scale adjustment, which can be approximated analytically, is applied. To the best of the authors' knowledge, this work is the first to quantize the weights, activations, as well as a substantial volume of the gradients stream, in all layers (including batch normalization) to 8-bit while showing state-of-the-art results over the ImageNet-1K dataset.

References in corpus (15)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Deep Learning with Limited Numerical Precision
• One weird trick for parallelizing convolutional neural networks
• Binarized Neural Networks
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Compressing Neural Networks with the Hashing Trick
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Towards Accurate Binary Convolutional Neural Network
• Convolutional Neural Networks using Logarithmic Data Representation
• Training and Inference with Integers in Deep Neural Networks
• WRPN: Wide Reduced-Precision Networks
• Mixed Precision Training of Convolutional Neural Networks using Integer Operations
• Norm matters: efficient and accurate normalization schemes in deep networks

Cited by in corpus (45)

• Benchmarking TPU, GPU, and CPU Platforms for Deep Learning
• DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks
• Rotated Binary Neural Network
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey
• Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks
• ShiftAddNet: A Hardware-Inspired Deep Network
• NITI: Training Integer Neural Networks Using Integer-only Arithmetic
• Low-Memory Neural Network Training: A Technical Report
• Lightweight Compression of Intermediate Neural Network Features for Collaborative Intelligence
• QGAN: Quantized Generative Adversarial Networks
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• A Panda? No, It's a Sloth: Slowdown Attacks on Adaptive Multi-Exit Neural Network Inference
• Few Shot Network Compression via Cross Distillation
• Taxonomy and Benchmarking of Precision-Scalable MAC Arrays Under Enhanced DNN Dataflow Representation
• Backprop with Approximate Activations for Memory-efficient Network Training
• Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks
• LeanConvNets: Low-cost Yet Effective Convolutional Neural Networks
• Mirage: An RNS-Based Photonic Accelerator for DNN Training
• Low Precision Decentralized Distributed Training over IID and non-IID Data
• CPT: Efficient Deep Neural Network Training via Cyclic Precision
• Memory Optimization for Deep Networks
• Sparse Weight Activation Training
• Training High-Performance and Large-Scale Deep Neural Networks with Full 8-bit Integers
• Neural Architecture Search using Property Guided Synthesis
• Shift-BNN: Highly-Efficient Probabilistic Bayesian Neural Network Training via Memory-Friendly Pattern Retrieving
• HadaNets: Flexible Quantization Strategies for Neural Networks
• Neural gradients are near-lognormal: improved quantized and sparse training
• A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off
• A 14uJ/Decision Keyword Spotting Accelerator with In-SRAM-Computing and On Chip Learning for Customization
• Low-Rank Training of Deep Neural Networks for Emerging Memory Technology
• A High-Performance Adaptive Quantization Approach for Edge CNN Applications
• OverQ: Opportunistic Outlier Quantization for Neural Network Accelerators
• n-hot: Efficient bit-level sparsity for powers-of-two neural network quantization
• NeRV: Neural Representations for Videos
• Neural Network Training on In-memory-computing Hardware with Radix-4 Gradients
• NaviSplit: Dynamic Multi-Branch Split DNNs for Efficient Distributed Autonomous Navigation
• AQD: Towards Accurate Fully-Quantized Object Detection
• AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networks
• "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization
• Prune Your Model Before Distill It
• NASA: Neural Architecture Search and Acceleration for Hardware Inspired Hybrid Networks
• InstantNet: Automated Generation and Deployment of Instantaneously Switchable-Precision Networks
• Batch Normalization Sampling
• Smoothed Differential Privacy
• SmartDeal: Re-Modeling Deep Network Weights for Efficient Inference and Training