Information Aggregation for Multi-Head Attention with Routing-by-Agreement
arXiv:1904.03100
Abstract
Multi-head attention is appealing for its ability to jointly extract different types of information from multiple representation subspaces. Concerning the information aggregation, a common practice is to use a concatenation followed by a linear transformation, which may not fully exploit the expressiveness of multi-head attention. In this work, we propose to improve the information aggregation for multi-head attention with a more powerful routing-by-agreement algorithm. Specifically, the routing algorithm iteratively updates the proportion of how much a part (i.e. the distinct information learned from a specific subspace) should be assigned to a whole (i.e. the final output representation), based on the agreement between parts and wholes. Experimental results on linguistic probing tasks and machine translation tasks prove the superiority of the advanced information aggregation over the standard linear transformation.
NAACL 2019
References in corpus (13)
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Convolutional Sequence to Sequence Learning
- A Structured Self-attentive Sentence Embedding
- Achieving Human Parity on Automatic Chinese to English News Translation
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- Capsules for Object Segmentation
- Capsule Network Performance on Complex Data
- Weighted Transformer Network for Machine Translation
- Information Aggregation via Dynamic Routing for Sequence Encoding
- Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architectures
- Exploiting Deep Representations for Neural Machine Translation
- Multi-Head Attention with Disagreement Regularization
- Convolutional Self-Attention Networks
Cited by in corpus (7)
- Convolutional Self-Attention Networks
- SesameBERT: Attention for Anywhere
- Generating Diverse Translation by Manipulating Multi-Head Attention
- Neuron Interaction Based Representation Composition for Neural Machine Translation
- Semantic Representation and Inference for NLP
- Unsupervised Neural Dialect Translation with Commonality and Diversity Modeling
- Capturing Argument Interaction in Semantic Role Labeling with Capsule Networks