Finding the Pillars of Strength for Multi-Head Attention

Jinjie Ni, Rui Mao, Zonglin Yang +2

Recent studies have revealed some issues of Multi-Head Attention (MHA), e.g., redundancy and over-parameterization. Specifically, the heads of MHA were originally designed to atten…