Rethinking Floating Point Overheads for Mixed Precision DNN Accelerators

Abstract

In this paper, we propose a mixed-precision convolution unit architecture which supports different integer and floating point (FP) precisions. The proposed architecture is based on low-bit inner product units and realizes higher precision based on temporal decomposition. We illustrate how to integrate FP computations on integer-based architecture and evaluate overheads incurred by FP arithmetic support. We argue that alignment and addition overhead for FP inner product can be significant since the maximum exponent difference could be up to 58 bits, which results into a large alignment logic. To address this issue, we illustrate empirically that no more than 26-bitproduct bits are required and up to 8-bit of alignment is sufficient in most inference cases. We present novel optimizations based on the above observations to reduce the FP arithmetic hardware overheads. Our empirical results, based on simulation and hardware implementation, show significant reduction in FP16 overhead. Over typical mixed precision implementation, the proposed architecture achieves area improvements of up to 25% in TFLOPS/mm2and up to 46% in TOPS/mm2with power efficiency improvements of up to 40% in TFLOPS/Wand up to 63% in TOPS/W.

Accepted to appear in 4th Conference on Machine Learning and Systems 2021

References in corpus (18)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• A guide to convolution arithmetic for deep learning
• Mixed Precision Training
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
• fairseq: A Fast, Extensible Toolkit for Sequence Modeling
• Mixed Precision Training of Convolutional Neural Networks using Integer Operations
• High-Accuracy Low-Precision Training
• Training DNNs with Hybrid Block Floating Point
• Quantization for Rapid Deployment of Deep Neural Networks
• AdaptivFloat: A Floating-point based Data Type for Resilient Deep Learning Inference
• Training Deep Neural Networks Using Posit Number System
• Mixed-Precision Quantized Neural Network with Progressively Decreasing Bitwidth For Image Classification and Object Detection