Relaxed Quantization for Discretized Neural Networks

Abstract

Neural network quantization has become an important research area due to its great impact on deployment of large models on resource constrained devices. In order to train networks that can be effectively discretized without loss of performance, we introduce a differentiable quantization procedure. Differentiability can be achieved by transforming continuous distributions over the weights and activations of the network to categorical distributions over the quantization grid. These are subsequently relaxed to continuous surrogates that can allow for efficient gradient-based optimization. We further show that stochastic rounding can be seen as a special case of the proposed approach and that under this formulation the quantization grid itself can also be optimized with gradient descent. We experimentally validate the performance of our method on MNIST, CIFAR 10 and Imagenet classification.

References in corpus (19)

• Adam: A Method for Stochastic Optimization
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Categorical Reparameterization with Gumbel-Softmax
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Ternary Weight Networks
• Trained Ternary Quantization
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Bayesian Compression for Deep Learning
• Training and Inference with Integers in Deep Neural Networks
• Model compression via distillation and quantization
• WRPN: Wide Reduced-Precision Networks
• Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
• Deep Convolutional Neural Network Inference with Floating-point Weights and Fixed-point Activations
• Deep Learning with Low Precision by Half-wave Gaussian Quantization
• Probabilistic Binary Neural Networks
• Low-Precision Floating-Point Schemes for Neural Network Training

Cited by in corpus (5)

• A Data and Compute Efficient Design for Limited-Resources Deep Learning
• Propagating Asymptotic-Estimated Gradients for Low Bitwidth Quantized Neural Networks
• Post-training Quantization with Multiple Points: Mixed Precision without Mixed Precision
• Accelerating Neural Network Inference by Overflow Aware Quantization
• Maximin Optimization for Binary Regression