Multi-Head Attention with Disagreement Regularization
arXiv:1810.10183
Abstract
Multi-head attention is appealing for the ability to jointly attend to information from different representation subspaces at different positions. In this work, we introduce a disagreement regularization to explicitly encourage the diversity among multiple attention heads. Specifically, we propose three types of disagreement regularization, which respectively encourage the subspace, the attended positions, and the output representation associated with each attention head to be different from other heads. Experimental results on widely-used WMT14 English-German and WMT17 Chinese-English translation tasks demonstrate the effectiveness and universality of the proposed approach.
arXiv commentEMNLP 2018
References in corpus (7)
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Convolutional Sequence to Sequence Learning
- Achieving Human Parity on Automatic Chinese to English News Translation
- Modeling Coverage for Neural Machine Translation
- Weighted Transformer Network for Machine Translation
- THUMT: An Open Source Toolkit for Neural Machine Translation
- Exploiting Deep Representations for Neural Machine Translation
Cited by in corpus (8)
- Syntax-Infused Transformer and BERT models for Machine Translation and Natural Language Understanding
- Context-Aware Self-Attention Networks
- Exploiting Deep Representations for Neural Machine Translation
- Convolutional Self-Attention Networks
- Information Aggregation for Multi-Head Attention with Routing-by-Agreement
- Generating Diverse Translation by Manipulating Multi-Head Attention
- Unsupervised Neural Dialect Translation with Commonality and Diversity Modeling
- Orthogonality Constrained Multi-Head Attention For Keyword Spotting