BiQGEMM: Matrix Multiplication with Lookup Table For Binary-Coding-based Quantized DNNs
arXiv:2005.09904
Abstract
The number of parameters in deep neural networks (DNNs) is rapidly increasing to support complicated tasks and to improve model accuracy. Correspondingly, the amount of computations and required memory footprint increase as well. Quantization is an efficient method to address such concerns by compressing DNNs such that computations can be simplified while required storage footprint is significantly reduced. Unfortunately, commercial CPUs and GPUs do not fully support quantization because only fixed data transfers (such as 32 bits) are allowed. As a result, even if weights are quantized into a few bits, CPUs and GPUs cannot access multiple quantized weights without memory bandwidth waste. Success of quantization in practice, hence, relies on an efficient computation engine design, especially for matrix multiplication that is a basic computation engine in most DNNs. In this paper, we propose a novel matrix multiplication method, called BiQGEMM, dedicated to quantized DNNs. BiQGEMM can access multiple quantized weights simultaneously in one instruction. In addition, BiQGEMM pre-computes intermediate results that are highly redundant when quantization leads to limited available computation space. Since pre-computed values are stored in lookup tables and reused, BiQGEMM achieves lower amount of overall computations. Our extensive experimental results show that BiQGEMM presents higher performance than conventional schemes when DNNs are quantized.
13 pages, 12 figures
References in corpus (17)
- SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
- Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- cuDNN: Efficient Primitives for Deep Learning
- A Survey of Model Compression and Acceleration for Deep Neural Networks
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- A Comparative Study on Transformer vs RNN in Speech Applications
- ERNIE: Enhanced Language Representation with Informative Entities
- Weighted Transformer Network for Machine Translation
- Understanding Back-Translation at Scale
- ERNIE 2.0: A Continual Pre-training Framework for Language Understanding
- And the Bit Goes Down: Revisiting the Quantization of Neural Networks
- Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Model
- LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks
- DeepTwist: Learning Model Compression via Occasional Weight Distortion
- Recovery guarantee of weighted low-rank approximation via alternating minimization
- Network Sketching: Exploiting Binary Structure in Deep CNNs