Dynamic Network Quantization for Efficient Video Inference

Abstract

Deep convolutional networks have recently achieved great success in video recognition, yet their practical realization remains a challenge due to the large amount of computational resources required to achieve robust recognition. Motivated by the effectiveness of quantization for boosting efficiency, in this paper, we propose a dynamic network quantization framework, that selects optimal precision for each frame conditioned on the input for efficient video recognition. Specifically, given a video clip, we train a very lightweight network in parallel with the recognition network, to produce a dynamic policy indicating which numerical precision to be used per frame in recognizing videos. We train both networks effectively using standard backpropagation with a loss to achieve both competitive performance and resource efficiency required for video recognition. Extensive experiments on four challenging diverse benchmark datasets demonstrate that our proposed approach provides significant savings in computation and memory usage while outperforming the existing state-of-the-art methods.

arXiv commentICCV 2021 Camera Ready Version

References in corpus (19)

• Distilling the Knowledge in a Neural Network
• Two-Stream Convolutional Networks for Action Recognition in Videos
• Categorical Reparameterization with Gumbel-Softmax
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Binarized Neural Networks
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Exploiting Feature and Class Relationships in Video Categorization with Regularized Deep Neural Networks
• Multi-Scale Dense Networks for Resource Efficient Image Classification
• Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
• Conditional Computation in Neural Networks for faster models
• Channel Gating Neural Networks
• More Is Less: Learning Efficient Video Representations by Big-Little Network and Depthwise Temporal Aggregation
• Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM
• Deciding How to Decide: Dynamic Routing in Artificial Neural Networks
• QKD: Quantization-aware Knowledge Distillation
• LiteEval: A Coarse-to-Fine Framework for Resource Efficient Video Recognition
• Deep Analysis of CNN-based Spatio-temporal Representations for Action Recognition
• Any-Precision Deep Neural Networks