Training Deep Neural Networks with 8-bit Floating Point Numbers

Abstract

The state-of-the-art hardware platforms for training Deep Neural Networks (DNNs) are moving from traditional single precision (32-bit) computations towards 16 bits of precision -- in large part due to the high energy efficiency and smaller bit storage associated with using reduced-precision representations. However, unlike inference, training with numbers represented with less than 16 bits has been challenging due to the need to maintain fidelity of the gradient computations during back-propagation. Here we demonstrate, for the first time, the successful training of DNNs using 8-bit floating point numbers while fully maintaining the accuracy on a spectrum of Deep Learning models and datasets. In addition to reducing the data and computation precision to 8 bits, we also successfully reduce the arithmetic precision for additions (used in partial product accumulation and weight updates) from 32 bits to 16 bits through the introduction of a number of key ideas including chunk-based accumulation and floating point stochastic rounding. The use of these novel techniques lays the foundation for a new generation of hardware training platforms with the potential for 2-4x improved throughput over today's systems.

NeurIPS 2018 (12 pages)

Cited by in corpus (30)

• Terminal Brain Damage: Exposing the Graceless Degradation in Deep Neural Networks Under Hardware Fault Attacks
• Mixed Precision Training With 8-bit Floating Point
• Towards Unified INT8 Training for Convolutional Neural Network
• A Statistical Framework for Low-bitwidth Training of Deep Neural Networks
• QPyTorch: A Low-Precision Arithmetic Simulation Framework
• EMPIR: Ensembles of Mixed Precision Deep Networks for Increased Robustness against Adversarial Attacks
• 5 Parallel Prism: A topology for pipelined implementations of convolutional neural networks using computational memory
• SmartExchange: Trading Higher-cost Memory Storage/Access for Lower-cost Computation
• Why is FPGA-GPU Heterogeneity the Best Option for Embedded Deep Neural Networks?
• Shift-BNN: Highly-Efficient Probabilistic Bayesian Neural Network Training via Memory-Friendly Pattern Retrieving
• WrapNet: Neural Net Inference with Ultra-Low-Resolution Arithmetic
• Adaptive Loss Scaling for Mixed Precision Training
• Adaptive Precision Training: Quantify Back Propagation in Neural Networks with Fixed-point Numbers
• Phoenix: A Low-Precision Floating-Point Quantization Oriented Architecture for Convolutional Neural Networks
• REPrune: Filter Pruning via Representative Election
• The Future of Computing: Bits + Neurons + Qubits
• CodeNet: Training Large Scale Neural Networks in Presence of Soft-Errors
• A Simple and Efficient Stochastic Rounding Method for Training Neural Networks in Low Precision
• Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks
• Distributed Deep Learning Strategies For Automatic Speech Recognition
• An FPGA Accelerated Method for Training Feed-forward Neural Networks Using Alternating Direction Method of Multipliers and LSMR
• A VM/Containerized Approach for Scaling TinyML Applications
• An Efficient Deep Learning Framework for Low Rate Massive MIMO CSI Reporting
• E2-Train: Training State-of-the-art CNNs with Over 80% Energy Savings
• An Analysis of Alternating Direction Method of Multipliers for Feed-forward Neural Networks
• Entropy-Based Modeling for Estimating Soft Errors Impact on Binarized Neural Network Inference
• All-You-Can-Fit 8-Bit Flexible Floating-Point Format for Accurate and Memory-Efficient Inference of Deep Neural Networks
• NEAT: A Framework for Automated Exploration of Floating Point Approximations
• Training and Inference for Integer-Based Semantic Segmentation Network
• Resource-Efficient Speech Mask Estimation for Multi-Channel Speech Enhancement