A General Survey on Attention Mechanisms in Deep Learning
arXiv:2203.14263 · doi:10.1109/TKDE.2021.3126456
Abstract
Attention is an important mechanism that can be employed for a variety of deep learning models across many different domains and tasks. This survey provides an overview of the most important attention mechanisms proposed in the literature. The various attention mechanisms are explained by means of a framework consisting of a general attention model, uniform notation, and a comprehensive taxonomy of attention mechanisms. Furthermore, the various measures for evaluating attention models are reviewed, and methods to characterize the structure of attention models based on the proposed framework are discussed. Last, future work in the field of attention models is considered.
arXiv comment20 pages, 11 figures
References in corpus (32)
- Neural Machine Translation by Jointly Learning to Align and Translate
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Attention U-Net: Learning Where to Look for the Pancreas
- TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
- Self-Attention Generative Adversarial Networks
- Attention-Based Models for Speech Recognition
- Bidirectional Attention Flow for Machine Comprehension
- A Structured Self-attentive Sentence Embedding
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- Linformer: Self-Attention with Linear Complexity
- Deep Audio-Visual Speech Recognition
- Attention is not Explanation
- On the Automatic Generation of Medical Imaging Reports
- Unsupervised Attention-guided Image to Image Translation
- BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer
- Temporal Attention augmented Bilinear Network for Financial Time-Series Data Analysis
- Attention-based Graph Neural Network for Semi-supervised Learning
- Multi-Pointer Co-Attention Networks for Recommendation
- Efficient Transformers: A Survey
- Synthesizer: Rethinking Self-Attention in Transformer Models
- HA-CCN: Hierarchical Attention-based Crowd Counting Network
- Residual Attention U-Net for Automated Multi-Class Segmentation of COVID-19 Chest CT Images
- Self-Attention Networks for Connectionist Temporal Classification in Speech Recognition
- Speech Emotion Recognition Using Multi-hop Attention Mechanism
- Survey on the attention based RNN model and its applications in computer vision
- Hierarchical Attention Network for Action Recognition in Videos
- A Hierarchical Attention Model for Social Contextual Image Recommendation
- Multi-level Attention Model for Weakly Supervised Audio Classification
- Attention Models in Graphs: A Survey
- Left-Center-Right Separated Neural Network for Aspect-based Sentiment Analysis with Rotatory Attention
- Iterative Alternating Neural Attention for Machine Reading
- Multi-hop Reading Comprehension across Multiple Documents by Reasoning over Heterogeneous Graphs