Efficient Execution of Quantized Deep Learning Models: A Compiler Approach

Abstract

A growing number of applications implement predictive functions using deep learning models, which require heavy use of compute and memory. One popular technique for increasing resource efficiency is 8-bit integer quantization, in which 32-bit floating point numbers (fp32) are represented using shorter 8-bit integer numbers. Although deep learning frameworks such as TensorFlow, TFLite, MXNet, and PyTorch enable developers to quantize models with only a small drop in accuracy, they are not well suited to execute quantized models on a variety of hardware platforms. For example, TFLite is optimized to run inference on ARM CPU edge devices but it does not have efficient support for Intel CPUs and Nvidia GPUs. In this paper, we address the challenges of executing quantized deep learning models on diverse hardware platforms by proposing an augmented compiler approach. A deep learning compiler such as Apache TVM can enable the efficient execution of model from various frameworks on various targets. Many deep learning compilers today, however, are designed primarily for fp32 computation and cannot optimize a pre-quantized INT8 model. To address this issue, we created a new dialect called Quantized Neural Network (QNN) that extends the compiler's internal representation with a quantization context. With this quantization context, the compiler can generate efficient code for pre-quantized models on various hardware platforms. As implemented in Apache TVM, we observe that the QNN-augmented deep learning compiler achieves speedups of 2.35x, 2.15x, 1.35x and 1.40x on Intel Xeon Cascade Lake CPUs, Nvidia Tesla T4 GPUs, ARM Raspberry Pi3 and Pi4 respectively against well optimized fp32 execution, and comparable performance to the state-of-the-art framework-specific solutions.

References in corpus (9)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Deep Residual Learning for Image Recognition
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• cuDNN: Efficient Primitives for Deep Learning
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• Fixed Point Quantization of Deep Convolutional Networks
• Glow: Graph Lowering Compiler Techniques for Neural Networks
• Intel nGraph: An Intermediate Representation, Compiler, and Executor for Deep Learning
• Stripe: Tensor Compilation via the Nested Polyhedral Model

Cited by in corpus (7)

• UNIT: Unifying Tensorized Instruction Compilation
• Quantune: Post-training Quantization of Convolutional Neural Networks using Extreme Gradient Boosting for Fast Deployment
• Computer Vision Model Compression Techniques for Embedded Systems: A Survey
• HAWQV3: Dyadic Neural Network Quantization
• Pre-Quantized Deep Learning Models Codified in ONNX to Enable Hardware/Software Co-Design
• Automated Backend-Aware Post-Training Quantization
• Understanding Cache Boundness of ML Operators on ARM Processors