Revisiting BFloat16 Training

Abstract

State-of-the-art generic low-precision training algorithms use a mix of 16-bit and 32-bit precision, creating the folklore that 16-bit hardware compute units alone are not enough to maximize model accuracy. As a result, deep learning accelerators are forced to support both 16-bit and 32-bit floating-point units (FPUs), which is more costly than only using 16-bit FPUs for hardware design. We ask: can we train deep learning models only with 16-bit floating-point units, while still matching the model accuracy attained by 32-bit training? Towards this end, we study 16-bit-FPU training on the widely adopted BFloat16 unit. While these units conventionally use nearest rounding to cast output to 16-bit precision, we show that nearest rounding for model weight updates often cancels small updates, which degrades the convergence and model accuracy. Motivated by this, we study two simple techniques well-established in numerical analysis, stochastic rounding and Kahan summation, to remedy the model accuracy degradation in 16-bit-FPU training. We demonstrate that these two techniques can enable up to 7% absolute validation accuracy gain in 16-bit-FPU training. This leads to 0.1% lower to 0.2% higher validation accuracy compared to 32-bit training across seven deep learning applications.

References in corpus (17)

• Decoupled Weight Decay Regularization
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Mixed Precision Training
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Pointer Sentinel Mixture Models
• Deep Learning Recommendation Model for Personalization and Recommendation Systems
• Training deep neural networks with low precision multiplications
• Training Deep Neural Networks with 8-bit Floating Point Numbers
• Taming the Wild: A Unified Analysis of Hogwild!-Style Algorithms
• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
• A Study of BFLOAT16 for Deep Learning Training
• Recurrent Neural Networks With Limited Numerical Precision
• QPyTorch: A Low-Precision Arithmetic Simulation Framework

Cited by in corpus (1)

• Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision