Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Model

Abstract

In this work, we quantize a trained Transformer machine language translation model leveraging INT8/VNNI instructions in the latest Intel${}^{\circledR }$ Xeon${}^{\circledR }$ Cascade Lake processors to improve inference performance while maintaining less than 0.5$\%$ drop in accuracy. To the best of our knowledge, this is the first attempt in the industry to quantize the Transformer model. This has high impact as it clearly demonstrates the various complexities of quantizing the language translation model. We present novel quantization techniques directly in TensorFlow to opportunistically replace 32-bit floating point (FP32) computations with 8-bit integers (INT8) and transform the FP32 computational graph. We also present a bin-packing parallel batching technique to maximize CPU utilization. Overall, our optimizations with INT8/VNNI deliver 1.5X improvement over the best FP32 performance. Furthermore, it reveals the opportunities and challenges to boost performance of quantized deep learning inference and establishes best practices to run inference with high efficiency on Intel CPUs.

To appear at the Joint Workshop on On-Device Machine Learning & Compact Deep Neural Network Representations, 36th International Conference on Machine Learning, Long Beach, California, 2019

References in corpus (2)

• Ternary Neural Networks with Fine-Grained Quantization
• Generating Sentences by Editing Prototypes

Cited by in corpus (24)

• A Survey on Visual Transformer
• Q8BERT: Quantized 8Bit BERT
• Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
• DynaBERT: Dynamic BERT with Adaptive Width and Depth
• LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
• Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
• QKD: Quantization-aware Knowledge Distillation
• AdaptivFloat: A Floating-point based Data Type for Resilient Deep Learning Inference
• A Simplified Fully Quantized Transformer for End-to-end Speech Recognition
• I-BERT: Integer-only BERT Quantization
• VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inference
• Hardware Accelerator for Multi-Head Attention and Position-Wise Feed-Forward in the Transformer
• BiQGEMM: Matrix Multiplication with Lookup Table For Binary-Coding-based Quantized DNNs
• Efficient Softmax Approximation for Deep Neural Networks with Attention Mechanism
• Fast Interleaved Bidirectional Sequence Generation
• An Investigation on Different Underlying Quantization Schemes for Pre-trained Language Models
• n-hot: Efficient bit-level sparsity for powers-of-two neural network quantization
• On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers
• Understanding and Overcoming the Challenges of Efficient Transformer Quantization
• A Survey on Green Deep Learning
• The NiuTrans System for WNGT 2020 Efficiency Task
• The NiuTrans System for the WMT21 Efficiency Task
• Towards Fully 8-bit Integer Inference for the Transformer Model
• Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale