An Empirical Study of Spatial Attention Mechanisms in Deep Networks

Abstract

Attention mechanisms have become a popular component in deep neural networks, yet there has been little examination of how different influencing factors and methods for computing attention from these factors affect performance. Toward a better general understanding of attention mechanisms, we present an empirical study that ablates various spatial attention elements within a generalized attention formulation, encompassing the dominant Transformer attention as well as the prevalent deformable convolution and dynamic convolution modules. Conducted on a variety of applications, the study yields significant findings about spatial attention in deep networks, some of which run counter to conventional understanding. For example, we find that the query and key content comparison in Transformer attention is negligible for self-attention, but vital for encoder-decoder attention. A proper combination of deformable convolution with key content only saliency achieves the best accuracy-efficiency tradeoff in self-attention. Our results suggest that there exists much room for improvement in the design of attention mechanisms.

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• A Deep Reinforced Model for Abstractive Summarization
• OCNet: Object Context Network for Scene Parsing
• Attention is not Explanation
• CCNet: Criss-Cross Attention for Semantic Segmentation
• Pay Less Attention with Lightweight and Dynamic Convolutions
• Dual Attention Network for Scene Segmentation
• Long Short-Term Memory-Networks for Machine Reading
• Deformable ConvNets v2: More Deformable, Better Results
• Massive Exploration of Neural Machine Translation Architectures
• A Decomposable Attention Model for Natural Language Inference
• Neural Machine Translation with Supervised Attention
• What does Attention in Neural Machine Translation Pay Attention to?
• Interpreting Recurrent and Attention-Based Neural Models: a Case Study on Natural Language Inference
• An Analysis of Attention Mechanisms: The Case of Word Sense Disambiguation in Neural Machine Translation

Cited by in corpus (11)

• Attention Mechanisms in Computer Vision: A Survey
• MMDetection: Open MMLab Detection Toolbox and Benchmark
• Stand-Alone Self-Attention in Vision Models
• Dynamic Neural Networks: A Survey
• Attentive Group Equivariant Convolutional Networks
• Empirical Upper Bound in Object Detection and More
• LiDAR-based Online 3D Video Object Detection with Graph-based Message Passing and Spatiotemporal Transformer Attention
• VIPriors 1: Visual Inductive Priors for Data-Efficient Deep Learning Challenges
• LIP: Local Importance-based Pooling
• Global Context Networks
• Follow the Attention: Combining Partial Pose and Object Motion for Fine-Grained Action Detection