Training DNNs with Hybrid Block Floating Point

Abstract

The wide adoption of DNNs has given birth to unrelenting computing requirements, forcing datacenter operators to adopt domain-specific accelerators to train them. These accelerators typically employ densely packed full precision floating-point arithmetic to maximize performance per area. Ongoing research efforts seek to further increase that performance density by replacing floating-point with fixed-point arithmetic. However, a significant roadblock for these attempts has been fixed point's narrow dynamic range, which is insufficient for DNN training convergence. We identify block floating point (BFP) as a promising alternative representation since it exhibits wide dynamic range and enables the majority of DNN operations to be performed with fixed-point logic. Unfortunately, BFP alone introduces several limitations that preclude its direct applicability. In this work, we introduce HBFP, a hybrid BFP-FP approach, which performs all dot products in BFP and other operations in floating point. HBFP delivers the best of both worlds: the high accuracy of floating point at the superior hardware density of fixed point. For a wide variety of models, we show that HBFP matches floating point's accuracy while enabling hardware implementations that deliver up to 8.5x higher throughput.

9 pages, 3 figures. Accepted in Neural Information Processing Systems 2018 (NeurIPS 2018)

References in corpus (4)

• Deep Learning with Limited Numerical Precision
• Mixed Precision Training
• Trained Ternary Quantization
• Regularizing and Optimizing LSTM Language Models

Cited by in corpus (8)

• Scaling Distributed Machine Learning with In-Network Aggregation
• TensorDash: Exploiting Sparsity to Accelerate Deep Neural Network Training and Inference
• DSConv: Efficient Convolution Operator
• A Statistical Framework for Low-bitwidth Training of Deep Neural Networks
• Rethinking Floating Point Overheads for Mixed Precision DNN Accelerators
• 8-bit Optimizers via Block-wise Quantization
• An FPGA Accelerated Method for Training Feed-forward Neural Networks Using Alternating Direction Method of Multipliers and LSMR
• Boosted and Differentially Private Ensembles of Decision Trees