Flash Multi-Head Feed-Forward Network

Minshen Zhang, Xiang Hu, Jianguo Li +2

We explore Multi-Head FFN (MH-FFN) as a replacement of FFN in the Transformer architecture, motivated by the structural similarity between single-head attention and FFN. While mult…