-Nets: Double Attention Networks
arXiv:1810.11579
Abstract
Learning to capture long-range relations is fundamental to image/video recognition. Existing CNN models generally rely on increasing depth to model such relations which is highly inefficient. In this work, we propose the "double attention block", a novel component that aggregates and propagates informative global features from the entire spatio-temporal space of input images/videos, enabling subsequent convolution layers to access features from the entire space efficiently. The component is designed with a double attention mechanism in two steps, where the first step gathers features from the entire space into a compact set through second-order attention pooling and the second step adaptively selects and distributes features to each location via another attention. The proposed double attention block is easy to adopt and can be plugged into existing deep neural networks conveniently. We conduct extensive ablation studies and experiments on both image and video recognition tasks for evaluating its performance. On the image recognition task, a ResNet-50 equipped with our double attention blocks outperforms a much larger ResNet-152 architecture on ImageNet-1k dataset with over 40% less the number of parameters and less FLOPs. On the action recognition task, our proposed model achieves the state-of-the-art results on the Kinetics and UCF-101 datasets with significantly higher efficiency than recent works.
arXiv commentAccepted at NIPS 2018
References in corpus (6)
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- The Kinetics Human Action Video Dataset
- MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
- ConvNet Architecture Search for Spatiotemporal Feature Learning
- ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
- Is Second-order Information Helpful for Large-scale Visual Recognition?
Cited by in corpus (4)
- VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
- Space-time Mixing Attention for Video Transformer
- Contrastive Learning for Local and Global Learning MRI Reconstruction
- Enhancing Transformer for Video Understanding Using Gated Multi-Level Attention and Temporal Adversarial Training