Repulsive Attention: Rethinking Multi-head Attention as Bayesian Inference

Bang An, Jie Lyu, Zhenyi Wang +6

The neural attention mechanism plays an important role in many natural language processing applications. In particular, the use of multi-head attention extends single-head attentio…