Dynamic Convolution: Attention over Convolution Kernels
arXiv:1912.03458
Abstract
Light-weight convolutional neural networks (CNNs) suffer performance degradation as their low computational budgets constrain both the depth (number of convolution layers) and the width (number of channels) of CNNs, resulting in limited representation capability. To address this issue, we present Dynamic Convolution, a new design that increases model complexity without increasing the network depth or width. Instead of using a single convolution kernel per layer, dynamic convolution aggregates multiple parallel convolution kernels dynamically based upon their attentions, which are input dependent. Assembling multiple kernels is not only computationally efficient due to the small kernel size, but also has more representation power since these kernels are aggregated in a non-linear way via attention. By simply using dynamic convolution for the state-of-the-art architecture MobileNetV3-Small, the top-1 accuracy of ImageNet classification is boosted by 2.9% with only 4% additional FLOPs and 2.9 AP gain is achieved on COCO keypoint detection.
arXiv commentCVPR 2020 (Oral)
References in corpus (10)
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
- Neural Architecture Search with Reinforcement Learning
- DARTS: Differentiable Architecture Search
- AMC: AutoML for Model Compression and Acceleration on Mobile Devices
- Trained Ternary Quantization
- CondConv: Conditionally Parameterized Convolutions for Efficient Inference
- Single Path One-Shot Neural Architecture Search with Uniform Sampling
- Multi-Scale Dense Networks for Resource Efficient Image Classification
- Shift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutions