QuantNet: Learning to Quantize by Learning within Fully Differentiable Framework

Abstract

Despite the achievements of recent binarization methods on reducing the performance degradation of Binary Neural Networks (BNNs), gradient mismatching caused by the Straight-Through-Estimator (STE) still dominates quantized networks. This paper proposes a meta-based quantizer named QuantNet, which utilizes a differentiable sub-network to directly binarize the full-precision weights without resorting to STE and any learnable gradient estimators. Our method not only solves the problem of gradient mismatching, but also reduces the impact of discretization errors, caused by the binarizing operation in the deployment, on performance. Generally, the proposed algorithm is implemented within a fully differentiable framework, and is easily extended to the general network quantization with any bits. The quantitative experiments on CIFAR-100 and ImageNet demonstrate that QuantNet achieves the signifficant improvements comparing with previous binarization methods, and even bridges gaps of accuracies between binarized models and full-precision models.

Accepted for publication in ECCV Workshop 2020

References in corpus (10)

• Distilling the Knowledge in a Neural Network
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• On the Convergence of Adam and Beyond
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets
• Mixed Precision DNNs: All you need is a good parametrization
• Self-Binarizing Networks
• Blended Coarse Gradient Descent for Full Quantization of Deep Neural Networks