Learning Sparse & Ternary Neural Networks with Entropy-Constrained Trained Ternarization (EC2T)
arXiv:2004.01077 · doi:10.1109/CVPRW50498.2020.00369
Abstract
Deep neural networks (DNN) have shown remarkable success in a variety of machine learning applications. The capacity of these models (i.e., number of parameters), endows them with expressive power and allows them to reach the desired performance. In recent years, there is an increasing interest in deploying DNNs to resource-constrained devices (i.e., mobile devices) with limited energy, memory, and computational budget. To address this problem, we propose Entropy-Constrained Trained Ternarization (EC2T), a general framework to create sparse and ternary neural networks which are efficient in terms of storage (e.g., at most two binary-masks and two full-precision values are required to save a weight matrix) and computation (e.g., MAC operations are reduced to a few accumulations plus two multiplications). This approach consists of two steps. First, a super-network is created by scaling the dimensions of a pre-trained model (i.e., its width and depth). Subsequently, this super-network is simultaneously pruned (using an entropy constraint) and quantized (that is, ternary values are assigned layer-wise) in a training process, resulting in a sparse and ternary network representation. We validate the proposed approach in CIFAR-10, CIFAR-100, and ImageNet datasets, showing its effectiveness in image classification tasks.
arXiv commentProceedings of the CVPR'20 Joint Workshop on Efficient Deep Learning in Computer Vision. Code is available at https://github.com/d-becking/efficientCNNs
References in corpus (15)
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- Ternary Weight Networks
- Trained Ternary Quantization
- The State of Sparsity in Deep Neural Networks
- Bayesian Compression for Deep Learning
- Variational Dropout Sparsifies Deep Neural Networks
- Learning Sparse Neural Networks through Regularization
- DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks
- Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM
- Towards the Limit of Network Quantization
- Deep Neural Network Compression with Single and Multiple Level Quantization
- Improved Bayesian Compression