Iteratively Training Look-Up Tables for Network Quantization
arXiv:1911.04951 · doi:10.1109/JSTSP.2020.3005030
Abstract
Operating deep neural networks (DNNs) on devices with limited resources requires the reduction of their memory as well as computational footprint. Popular reduction methods are network quantization or pruning, which either reduce the word length of the network parameters or remove weights from the network if they are not needed. In this article we discuss a general framework for network reduction which we call `Look-Up Table Quantization` (LUT-Q). For each layer, we learn a value dictionary and an assignment matrix to represent the network weights. We propose a special solver which combines gradient descent and a one-step k-means update to learn both the value dictionaries and assignment matrices iteratively. This method is very flexible: by constraining the value dictionary, many different reduction problems such as non-uniform network quantization, training of multiplierless networks, network pruning or simultaneous quantization and pruning can be implemented without changing the solver. This flexibility of the LUT-Q method allows us to use the same method to train networks for different hardware capabilities.
arXiv commentCopyright 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works
References in corpus (17)
- Adam: A Method for Stochastic Optimization
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Distilling the Knowledge in a Neural Network
- Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
- BinaryConnect: Training Deep Neural Networks with binary weights during propagations
- DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
- Ternary Weight Networks
- Compressing Neural Networks with the Hashing Trick
- Bayesian Compression for Deep Learning
- Learned Step Size Quantization
- WRPN: Wide Reduced-Precision Networks
- Soft Weight-Sharing for Neural Network Compression
- Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets
- Mixed Precision DNNs: All you need is a good parametrization
- Sharing Residual Units Through Collective Tensor Factorization in Deep Neural Networks
- Iteratively Training Look-Up Tables for Network Quantization