Multi-Head Attention with Disagreement Regularization

Jian Li, Zhaopeng Tu, Baosong Yang +2

Multi-head attention is appealing for the ability to jointly attend to information from different representation subspaces at different positions. In this work, we introduce a disa…