A Study of BFLOAT16 for Deep Learning Training

Abstract

This paper presents the first comprehensive empirical study demonstrating the efficacy of the Brain Floating Point (BFLOAT16) half-precision format for Deep Learning training across image classification, speech recognition, language modeling, generative networks and industrial recommendation systems. BFLOAT16 is attractive for Deep Learning training for two reasons: the range of values it can represent is the same as that of IEEE 754 floating-point format (FP32) and conversion to/from FP32 is simple. Maintaining the same range as FP32 is important to ensure that no hyper-parameter tuning is required for convergence; e.g., IEEE 754 compliant half-precision floating point (FP16) requires hyper-parameter tuning. In this paper, we discuss the flow of tensors and various key operations in mixed precision training, and delve into details of operations, such as the rounding modes for converting FP32 tensors to BFLOAT16. We have implemented a method to emulate BFLOAT16 operations in Tensorflow, Caffe2, IntelCaffe, and Neon for our experiments. Our results show that deep learning training using BFLOAT16 tensors achieves the same state-of-the-art (SOTA) results across domains as FP32 tensors in the same number of iterations and with no changes to hyper-parameters.

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Mixed Precision Training
• Deep Learning Recommendation Model for Personalization and Recommendation Systems
• Training deep neural networks with low precision multiplications
• 8-Bit Approximations for Parallelism in Deep Learning
• Ternary Neural Networks with Fine-Grained Quantization
• Automatic Full Compilation of Julia Programs and ML Models to Cloud TPUs
• High Performance Monte Carlo Simulation of Ising Model on TPU Clusters

Cited by in corpus (20)

• Hardware Acceleration of Sparse and Irregular Tensor Computations of ML Models: A Survey and Insights
• TensorDash: Exploiting Sparsity to Accelerate Deep Neural Network Training and Inference
• A TensorFlow Simulation Framework for Scientific Computing of Fluid Flows on Tensor Processing Units
• Degree-Quant: Quantization-Aware Training for Graph Neural Networks
• First-Generation Inference Accelerator Deployment at Facebook
• Mixed-Precision Embedding Using a Cache
• DarKnight: A Data Privacy Scheme for Training and Inference of Deep Neural Networks
• Searching for Winograd-aware Quantized Networks
• Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision
• Deep Learning Training on the Edge with Low-Precision Posits
• Revisiting BFloat16 Training
• Adaptive Loss Scaling for Mixed Precision Training
• Learning compositional functions via multiplicative weight updates
• Neural gradients are near-lognormal: improved quantized and sparse training
• Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks
• All-You-Can-Fit 8-Bit Flexible Floating-Point Format for Accurate and Memory-Efficient Inference of Deep Neural Networks
• RLIBM-ALL: A Novel Polynomial Approximation Method to Produce Correctly Rounded Results for Multiple Representations and Rounding Modes
• Representation range needs for 16-bit neural network training
• Reducing Data Motion to Accelerate the Training of Deep Neural Networks
• Deep Neural Network Training without Multiplications