A General Survey on Attention Mechanisms in Deep Learning

Abstract

Attention is an important mechanism that can be employed for a variety of deep learning models across many different domains and tasks. This survey provides an overview of the most important attention mechanisms proposed in the literature. The various attention mechanisms are explained by means of a framework consisting of a general attention model, uniform notation, and a comprehensive taxonomy of attention mechanisms. Furthermore, the various measures for evaluating attention models are reviewed, and methods to characterize the structure of attention models based on the proposed framework are discussed. Last, future work in the field of attention models is considered.

20 pages, 11 figures

References in corpus (9)

• TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
• Attention-Based Models for Speech Recognition
• A Structured Self-attentive Sentence Embedding
• Linformer: Self-Attention with Linear Complexity
• Attention is not Explanation
• Residual Attention U-Net for Automated Multi-Class Segmentation of COVID-19 Chest CT Images
• Self-Attention Networks for Connectionist Temporal Classification in Speech Recognition
• Hierarchical Attention Network for Action Recognition in Videos
• Multi-hop Reading Comprehension across Multiple Documents by Reasoning over Heterogeneous Graphs