Weight Normalization based Quantization for Deep Neural Network Compression

Abstract

With the development of deep neural networks, the size of network models becomes larger and larger. Model compression has become an urgent need for deploying these network models to mobile or embedded devices. Model quantization is a representative model compression technique. Although a lot of quantization methods have been proposed, many of them suffer from a high quantization error caused by a long-tail distribution of network weights. In this paper, we propose a novel quantization method, called weight normalization based quantization (WNQ), for model compression. WNQ adopts weight normalization to avoid the long-tail distribution of network weights and subsequently reduces the quantization error. Experiments on CIFAR-100 and ImageNet show that WNQ can outperform other baselines to achieve state-of-the-art performance.

10 pages, 5 figures

References in corpus (11)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Binarized Neural Networks
• Ternary Weight Networks
• Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM
• Heterogeneous Bitwidth Binarization in Convolutional Neural Networks
• Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration
• Synaptic Strength For Convolutional Neural Network
• Regularizing Activation Distribution for Training Binarized Deep Networks
• Simultaneously Optimizing Weight and Quantizer of Ternary Neural Network using Truncated Gaussian Approximation

Cited by in corpus (4)

• Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks
• Quantized Convolutional Neural Networks Through the Lens of Partial Differential Equations
• SQWA: Stochastic Quantized Weight Averaging for Improving the Generalization Capability of Low-Precision Deep Neural Networks
• GradFreeBits: Gradient Free Bit Allocation for Dynamic Low Precision Neural Networks