AdaptivFloat: A Floating-point based Data Type for Resilient Deep Learning Inference

Abstract

Conventional hardware-friendly quantization methods, such as fixed-point or integer, tend to perform poorly at very low word sizes as their shrinking dynamic ranges cannot adequately capture the wide data distributions commonly seen in sequence transduction models. We present AdaptivFloat, a floating-point inspired number representation format for deep learning that dynamically maximizes and optimally clips its available dynamic range, at a layer granularity, in order to create faithful encoding of neural network parameters. AdaptivFloat consistently produces higher inference accuracies compared to block floating-point, uniform, IEEE-like float or posit encodings at very low precision ($\le$ 8-bit) across a diverse set of state-of-the-art neural network topologies. And notably, AdaptivFloat is seen surpassing baseline FP32 performance by up to +0.3 in BLEU score and -0.75 in word error rate at weight bit widths that are $\le$ 8-bit. Experimental results on a deep neural network (DNN) hardware accelerator, exploiting AdaptivFloat logic in its computational datapath, demonstrate per-operation energy and area that is 0.9$\times$ and 1.14$\times$, respectively, that of equivalent bit width integer-based accelerator variants.

arXiv comment10 pages

References in corpus (13)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Deep Residual Learning for Image Recognition
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Attention-Based Models for Speech Recognition
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Deep Learning with Limited Numerical Precision
• Trained Ternary Quantization
• Convolutional Neural Networks using Logarithmic Data Representation
• WRPN: Wide Reduced-Precision Networks
• Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Model
• LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks
• Quantized Convolutional Neural Networks for Mobile Devices

Cited by in corpus (6)

• EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inference
• Elastic Significant Bit Quantization and Acceleration for Deep Neural Networks
• Rethinking Floating Point Overheads for Mixed Precision DNN Accelerators
• A Survey of FPGA-Based Robotic Computing
• Quantized Neural Network Inference with Precision Batching
• Analyzing and Improving Fault Tolerance of Learning-Based Navigation Systems