Context-Aware Self-Attention Networks
arXiv:1902.05766
Abstract
Self-attention model have shown its flexibility in parallel computation and the effectiveness on modeling both long- and short-term dependencies. However, it calculates the dependencies between representations without considering the contextual information, which have proven useful for modeling dependencies among neural representations in various natural language tasks. In this work, we focus on improving self-attention networks through capturing the richness of context. To maintain the simplicity and flexibility of the self-attention networks, we propose to contextualize the transformations of the query and key layers, which are used to calculates the relevance between elements. Specifically, we leverage the internal representations that embed both global and deep contexts, thus avoid relying on external resources. Experimental results on WMT14 English-German and WMT17 Chinese-English translation tasks demonstrate the effectiveness and universality of the proposed methods. Furthermore, we conducted extensive analyses to quantity how the context vectors participate in the self-attention model.
AAAI 2019
References in corpus (19)
- Neural Machine Translation by Jointly Learning to Align and Translate
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- A Structured Self-attentive Sentence Embedding
- Effective Approaches to Attention-based Neural Machine Translation
- Achieving Human Parity on Automatic Chinese to English News Translation
- Modeling Coverage for Neural Machine Translation
- Contextual Action Recognition with R*CNN
- Evaluating Discourse Phenomena in Neural Machine Translation
- Beyond Local Search: Tracking Objects Everywhere with Instance-Specific Proposals
- Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architectures
- Exploiting Cross-Sentence Context for Neural Machine Translation
- Exploiting Deep Representations for Neural Machine Translation
- Accelerating Neural Transformer via an Average Attention Network
- Multi-Head Attention with Disagreement Regularization
- Convolutional Self-Attention Networks
- Towards Bidirectional Hierarchical Representations for Attention-Based Neural Machine Translation
- Convolutional Self-Attention Network
- Learning to Jointly Translate and Predict Dropped Pronouns with a Shared Reconstruction Mechanism