Loss-aware Weight Quantization of Deep Networks

Abstract

The huge size of deep networks hinders their use in small computing devices. In this paper, we consider compressing the network by weight quantization. We extend a recently proposed loss-aware weight binarization scheme to ternarization, with possibly different scaling parameters for the positive and negative weights, and m-bit (where m > 2) quantization. Experiments on feedforward and recurrent neural networks show that the proposed scheme outperforms state-of-the-art weight quantization algorithms, and is as accurate (or even more accurate) than the full-precision network.

References in corpus (13)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Visualizing and Understanding Recurrent Networks
• Ternary Weight Networks
• Trained Ternary Quantization
• Tensorizing Neural Networks
• Convolutional Neural Networks using Logarithmic Data Representation
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Pylearn2: a machine learning research library
• Loss-aware Binarization of Deep Networks
• Neural Networks with Few Multiplications

Cited by in corpus (21)

• A Survey on Methods and Theories of Quantized Neural Networks
• MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning
• ProxQuant: Quantized Neural Networks via Proximal Operators
• Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets
• BinaryBERT: Pushing the Limit of BERT Quantization
• Learning Sparse Low-Precision Neural Networks With Learnable Regularization
• Adaptive Low-Precision Training for Embeddings in Click-Through Rate Prediction
• Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation
• FxP-QNet: A Post-Training Quantizer for the Design of Mixed Low-Precision DNNs with Dynamic Fixed-Point Representation
• Sparse Weight Activation Training
• TernaryBERT: Distillation-aware Ultra-low Bit BERT
• Effective Training of Convolutional Neural Networks with Low-bitwidth Weights and Activations
• NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization
• SinReQ: Generalized Sinusoidal Regularization for Low-Bitwidth Deep Quantized Training
• Adaptive Precision Training (AdaPT): A dynamic fixed point quantized training approach for DNNs
• Optimal Gradient Quantization Condition for Communication-Efficient Distributed Training
• Learning Recurrent Binary/Ternary Weights
• Balanced Binary Neural Networks with Gated Residual
• AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networks
• FATNN: Fast and Accurate Ternary Neural Networks
• Learning Quantized Neural Nets by Coarse Gradient Method for Non-linear Classification