Mixed Precision Training With 8-bit Floating Point

Abstract

Reduced precision computation for deep neural networks is one of the key areas addressing the widening compute gap driven by an exponential growth in model size. In recent years, deep learning training has largely migrated to 16-bit precision, with significant gains in performance and energy efficiency. However, attempts to train DNNs at 8-bit precision have met with significant challenges because of the higher precision and dynamic range requirements of back-propagation. In this paper, we propose a method to train deep neural networks using 8-bit floating point representation for weights, activations, errors, and gradients. In addition to reducing compute precision, we also reduced the precision requirements for the master copy of weights from 32-bit to 16-bit. We demonstrate state-of-the-art accuracy across multiple data sets (imagenet-1K, WMT16) and a broader set of workloads (Resnet-18/34/50, GNMT, Transformer) than previously reported. We propose an enhanced loss scaling method to augment the reduced subnormal range of 8-bit floating point for improved error propagation. We also examine the impact of quantization noise on generalization and propose a stochastic rounding technique to address gradient noise. As a result of applying all these techniques, we report slightly higher validation accuracy compared to full precision baseline.

References in corpus (9)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Mixed Precision Training
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Training Deep Neural Networks with 8-bit Floating Point Numbers
• Mixed Precision Training of Convolutional Neural Networks using Integer Operations
• High-Accuracy Low-Precision Training
• Mixed-Precision Training for NLP and Speech Recognition with OpenSeq2Seq

Cited by in corpus (7)

• Mixed-precision explicit stabilized Runge-Kutta methods for single- and multi-scale differential equations
• Effects of round-to-nearest and stochastic rounding in the numerical solution of the heat equation in low precision
• Deep Learning Training on the Edge with Low-Precision Posits
• An Automotive Case Study on the Limits of Approximation for Object Detection
• Adaptive Loss Scaling for Mixed Precision Training
• 8-bit Optimizers via Block-wise Quantization
• Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks