Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference

Abstract

Modern convolutional neural networks apply the same operations on every pixel in an image. However, not all image regions are equally important. To address this inefficiency, we propose a method to dynamically apply convolutions conditioned on the input image. We introduce a residual block where a small gating branch learns which spatial positions should be evaluated. These discrete gating decisions are trained end-to-end using the Gumbel-Softmax trick, in combination with a sparsity criterion. Our experiments on CIFAR, ImageNet and MPII show that our method has better focus on the region of interest and better accuracy than existing methods, at a lower computational complexity. Moreover, we provide an efficient CUDA implementation of our dynamic convolutions using a gather-scatter approach, achieving a significant improvement in inference speed with MobileNetV2 residual blocks. On human pose estimation, a task that is inherently spatially sparse, the processing speed is increased by 60% with no loss in accuracy.

CVPR 2020 (poster)

References in corpus (7)

• Distilling the Knowledge in a Neural Network
• FitNets: Hints for Thin Deep Nets
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Recurrent Models of Visual Attention
• Pruning Filters for Efficient ConvNets
• Submanifold Sparse Convolutional Networks
• Spatially-sparse convolutional neural networks

Cited by in corpus (11)

• Bringing AI To Edge: From Deep Learning's Perspective
• Dynamic Neural Networks: A Survey
• AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition
• QueryDet: Cascaded Sparse Query for Accelerating High-Resolution Small Object Detection
• Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition
• BlockCopy: High-Resolution Video Processing with Block-Sparse Feature Propagation and Online Policies
• SegBlocks: Block-Based Dynamic Resolution Networks for Real-Time Segmentation
• Dynamics-aware Adversarial Attack of Adaptive Neural Networks
• Learning Frequency-aware Dynamic Network for Efficient Super-Resolution
• CoDiNet: Path Distribution Modeling with Consistency and Diversity for Dynamic Routing
• Self-supervised Feature-Gate Coupling for Dynamic Network Pruning