DeepTwist: Learning Model Compression via Occasional Weight Distortion

Abstract

Model compression has been introduced to reduce the required hardware resources while maintaining the model accuracy. Lots of techniques for model compression, such as pruning, quantization, and low-rank approximation, have been suggested along with different inference implementation characteristics. Adopting model compression is, however, still challenging because the design complexity of model compression is rapidly increasing due to additional hyper-parameters and computation overhead in order to achieve a high compression ratio. In this paper, we propose a simple and efficient model compression framework called DeepTwist which distorts weights in an occasional manner without modifying the underlying training algorithms. The ideas of designing weight distortion functions are intuitive and straightforward given formats of compressed weights. We show that our proposed framework improves compression rate significantly for pruning, quantization, and low-rank approximation techniques while the efforts of additional retraining and/or hyper-parameter search are highly reduced. Regularization effects of DeepTwist are also reported.

References in corpus (8)

• Distilling the Knowledge in a Neural Network
• Recurrent Neural Network Regularization
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Trained Ternary Quantization
• The Loss Surfaces of Multilayer Networks
• Training and Inference with Integers in Deep Neural Networks
• Exploring Sparsity in Recurrent Neural Networks

Cited by in corpus (9)

• Learning Low-Rank Approximation for CNNs
• BiQGEMM: Matrix Multiplication with Lookup Table For Binary-Coding-based Quantized DNNs
• Network Pruning for Low-Rank Binary Indexing
• Compressing Language Models using Doped Kronecker Products
• A review of on-device fully neural end-to-end automatic speech recognition algorithms
• Weight Equalizing Shift Scaler-Coupled Post-training Quantization
• Stochastic Model Pruning via Weight Dropping Away and Back
• Structured Compression by Weight Encryption for Unstructured Pruning and Quantization
• Attention based on-device streaming speech recognition with large speech corpus