Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference

Abstract

Modern convolutional neural networks apply the same operations on every pixel in an image. However, not all image regions are equally important. To address this inefficiency, we propose a method to dynamically apply convolutions conditioned on the input image. We introduce a residual block where a small gating branch learns which spatial positions should be evaluated. These discrete gating decisions are trained end-to-end using the Gumbel-Softmax trick, in combination with a sparsity criterion. Our experiments on CIFAR, ImageNet and MPII show that our method has better focus on the region of interest and better accuracy than existing methods, at a lower computational complexity. Moreover, we provide an efficient CUDA implementation of our dynamic convolutions using a gather-scatter approach, achieving a significant improvement in inference speed with MobileNetV2 residual blocks. On human pose estimation, a task that is inherently spatially sparse, the processing speed is increased by 60% with no loss in accuracy.

arXiv commentCVPR 2020 (poster)

References in corpus (17)

• Distilling the Knowledge in a Neural Network
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Categorical Reparameterization with Gumbel-Softmax
• FitNets: Hints for Thin Deep Nets
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Recurrent Models of Visual Attention
• An Analysis of Deep Neural Network Models for Practical Applications
• Pruning Filters for Efficient ConvNets
• Action Recognition using Visual Attention
• Adaptive Computation Time for Recurrent Neural Networks
• Submanifold Sparse Convolutional Networks
• Dynamic Channel Pruning: Feature Boosting and Suppression
• Spatially-sparse convolutional neural networks
• Conditional Computation in Neural Networks for faster models
• Structured Transforms for Small-Footprint Deep Learning

Cited by in corpus (11)

• Bringing AI To Edge: From Deep Learning's Perspective
• Dynamic Neural Networks: A Survey
• AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition
• QueryDet: Cascaded Sparse Query for Accelerating High-Resolution Small Object Detection
• Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition
• BlockCopy: High-Resolution Video Processing with Block-Sparse Feature Propagation and Online Policies
• SegBlocks: Block-Based Dynamic Resolution Networks for Real-Time Segmentation
• Dynamics-aware Adversarial Attack of Adaptive Neural Networks
• Learning Frequency-aware Dynamic Network for Efficient Super-Resolution
• CoDiNet: Path Distribution Modeling with Consistency and Diversity for Dynamic Routing
• Self-supervised Feature-Gate Coupling for Dynamic Network Pruning