Training Deep Neural Networks with 8-bit Floating Point Numbers

Abstract

The state-of-the-art hardware platforms for training Deep Neural Networks (DNNs) are moving from traditional single precision (32-bit) computations towards 16 bits of precision -- in large part due to the high energy efficiency and smaller bit storage associated with using reduced-precision representations. However, unlike inference, training with numbers represented with less than 16 bits has been challenging due to the need to maintain fidelity of the gradient computations during back-propagation. Here we demonstrate, for the first time, the successful training of DNNs using 8-bit floating point numbers while fully maintaining the accuracy on a spectrum of Deep Learning models and datasets. In addition to reducing the data and computation precision to 8 bits, we also successfully reduce the arithmetic precision for additions (used in partial product accumulation and weight updates) from 32 bits to 16 bits through the introduction of a number of key ideas including chunk-based accumulation and floating point stochastic rounding. The use of these novel techniques lays the foundation for a new generation of hardware training platforms with the potential for 2-4x improved throughput over today's systems.

arXiv commentNeurIPS 2018 (12 pages)

Cited by in corpus (63)

• Digital Electronics and Analog Photonics for Convolutional Neural Networks (DEAP-CNNs)
• Terminal Brain Damage: Exposing the Graceless Degradation in Deep Neural Networks Under Hardware Fault Attacks
• TensorDash: Exploiting Sparsity to Accelerate Deep Neural Network Training and Inference
• Charged particle tracking via edge-classifying interaction networks
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey
• On-Device Machine Learning: An Algorithms and Learning Theory Perspective
• Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks
• Mixed Precision Training With 8-bit Floating Point
• TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learning
• COVID-MobileXpert: On-Device COVID-19 Patient Triage and Follow-up using Chest X-rays
• Towards Unified INT8 Training for Convolutional Neural Network
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Training large-scale ANNs on simulated resistive crossbar arrays
• A Statistical Framework for Low-bitwidth Training of Deep Neural Networks
• QPyTorch: A Low-Precision Arithmetic Simulation Framework
• EMPIR: Ensembles of Mixed Precision Deep Networks for Increased Robustness against Adversarial Attacks
• Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision
• CPT: Efficient Deep Neural Network Training via Cyclic Precision
• ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training
• Sparse Weight Activation Training
• Training High-Performance and Large-Scale Deep Neural Networks with Full 8-bit Integers
• Deep Learning Training on the Edge with Low-Precision Posits
• 5 Parallel Prism: A topology for pipelined implementations of convolutional neural networks using computational memory
• SmartExchange: Trading Higher-cost Memory Storage/Access for Lower-cost Computation
• Revisiting BFloat16 Training
• Why is FPGA-GPU Heterogeneity the Best Option for Embedded Deep Neural Networks?
• Shift-BNN: Highly-Efficient Probabilistic Bayesian Neural Network Training via Memory-Friendly Pattern Retrieving
• Adaptive Loss Scaling for Mixed Precision Training
• Training Deep Neural Networks Using Posit Number System
• WrapNet: Neural Net Inference with Ultra-Low-Resolution Arithmetic
• Neural gradients are near-lognormal: improved quantized and sparse training
• Phoenix: A Low-Precision Floating-Point Quantization Oriented Architecture for Convolutional Neural Networks
• Adaptive Precision Training: Quantify Back Propagation in Neural Networks with Fixed-point Numbers
• Learning compositional functions via multiplicative weight updates
• A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off
• Approximations in Deep Learning
• REPrune: Filter Pruning via Representative Election
• Trends and Advancements in Deep Neural Network Communication
• 8-bit Optimizers via Block-wise Quantization
• CodeNet: Training Large Scale Neural Networks in Presence of Soft-Errors
• The Future of Computing: Bits + Neurons + Qubits
• Better Schedules for Low Precision Training of Deep Neural Networks
• Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks
• A Simple and Efficient Stochastic Rounding Method for Training Neural Networks in Low Precision
• An FPGA Accelerated Method for Training Feed-forward Neural Networks Using Alternating Direction Method of Multipliers and LSMR
• A note on solving nonlinear optimization problems in variable precision
• Distributed Deep Learning Strategies For Automatic Speech Recognition
• Numerical Stability of DeepGOPlus Inference
• AQD: Towards Accurate Fully-Quantized Object Detection
• Randomization of Approximate Bilinear Computation for Matrix Multiplication
• Enabling Binary Neural Network Training on the Edge
• SIPA: A Simple Framework for Efficient Networks
• An Analysis of Alternating Direction Method of Multipliers for Feed-forward Neural Networks
• Entropy-Based Modeling for Estimating Soft Errors Impact on Binarized Neural Network Inference
• E2-Train: Training State-of-the-art CNNs with Over 80% Energy Savings
• All-You-Can-Fit 8-Bit Flexible Floating-Point Format for Accurate and Memory-Efficient Inference of Deep Neural Networks
• An Efficient Deep Learning Framework for Low Rate Massive MIMO CSI Reporting
• NEAT: A Framework for Automated Exploration of Floating Point Approximations
• A VM/Containerized Approach for Scaling TinyML Applications
• Training and Inference for Integer-Based Semantic Segmentation Network
• Resource-Efficient Speech Mask Estimation for Multi-Channel Speech Enhancement
• LightSeq2: Accelerated Training for Transformer-based Models on GPUs
• SmartDeal: Re-Modeling Deep Network Weights for Efficient Inference and Training