An Introductory Survey on Attention Mechanisms in NLP Problems
arXiv:1811.05544
Abstract
First derived from human intuition, later adapted to machine translation for automatic token alignment, attention mechanism, a simple method that can be used for encoding sequence data based on the importance score each element is assigned, has been widely applied to and attained significant improvement in various tasks in natural language processing, including sentiment classification, text summarization, question answering, dependency parsing, etc. In this paper, we survey through recent works and conduct an introductory summary of the attention mechanism in different NLP problems, aiming to provide our readers with basic knowledge on this widely used method, discuss its different variants for different tasks, explore its association with other techniques in machine learning, and examine methods for evaluating its performance.
9 pages
References in corpus (7)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Dynamic Memory Networks for Visual and Textual Question Answering
- Enriching Word Vectors with Subword Information
- DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding
- Multi-Level Structured Self-Attentions for Distantly Supervised Relation Extraction
- A Nested Attention Neural Hybrid Model for Grammatical Error Correction
- MARS: Memory Attention-Aware Recommender System