Improved Techniques for Quantizing Deep Networks with Adaptive Bit-Widths

Abstract

Quantizing deep networks with adaptive bit-widths is a promising technique for efficient inference across many devices and resource constraints. In contrast to static methods that repeat the quantization process and train different models for different constraints, adaptive quantization enables us to flexibly adjust the bit-widths of a single deep network during inference for instant adaptation in different scenarios. While existing research shows encouraging results on common image classification benchmarks, this paper investigates how to train such adaptive networks more effectively. Specifically, we present two novel techniques for quantizing deep neural networks with adaptive bit-widths of weights and activations. First, we propose a collaborative strategy to choose a high-precision teacher for transferring knowledge to the low-precision student while jointly optimizing the model with all bit-widths. Second, to effectively transfer knowledge, we develop a dynamic block swapping method by randomly replacing the blocks in the lower-precision student network with the corresponding blocks in the higher-precision teacher network. Extensive experiments on multiple image classification datasets including video classification benchmarks for the first time, well demonstrate the efficacy of our approach over state-of-the-art methods.

References in corpus (15)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distilling the Knowledge in a Neural Network
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Binarized Neural Networks
• Ternary Weight Networks
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Knowledge Distillation by On-the-Fly Native Ensemble
• Multi-Scale Dense Networks for Resource Efficient Image Classification
• Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
• Large scale distributed neural network training through online distillation
• Conditional Computation in Neural Networks for faster models
• AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning
• QKD: Quantization-aware Knowledge Distillation
• Interactive Knowledge Distillation