Automating Generation of Low Precision Deep Learning Operators
arXiv:1810.11066
Abstract
State of the art deep learning models have made steady progress in the fields of computer vision and natural language processing, at the expense of growing model sizes and computational complexity. Deploying these models on low power and mobile devices poses a challenge due to their limited compute capabilities and strict energy budgets. One solution that has generated significant research interest is deploying highly quantized models that operate on low precision inputs and weights less than eight bits, trading off accuracy for performance. These models have a significantly reduced memory footprint (up to 32x reduction) and can replace multiply-accumulates with bitwise operations during compute intensive convolution and fully connected layers. Most deep learning frameworks rely on highly engineered linear algebra libraries such as ATLAS or Intel's MKL to implement efficient deep learning operators. To date, none of the popular deep learning directly support low precision operators, partly due to a lack of optimized low precision libraries. In this paper we introduce a work flow to quickly generate high performance low precision deep learning operators for arbitrary precision that target multiple CPU architectures and include optimizations such as memory tiling and vectorization. We present an extensive case study on low power ARM Cortex-A53 CPU, and show how we can generate 1-bit, 2-bit convolutions with speedups up to 16x over an optimized 16-bit integer baseline and 2.3x better than handwritten implementations.
arXiv comment10 pages, 11 figures
References in corpus (11)
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
- XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
- Learning to Optimize Tensor Programs
- TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
- Deep Learning with Low Precision by Half-wave Gaussian Quantization
- Faster Population Counts Using AVX2 Instructions
- Heterogeneous Bitwidth Binarization in Convolutional Neural Networks
- High performance ultra-low-precision convolutions on mobile devices