Interactive Multi-Head Self-Attention with Linear Complexity

Hankyul Kang, Ming-Hsuan Yang, Jongbin Ryu

We propose an efficient interactive method for multi-head self-attention via decomposition. For existing methods using multi-head self-attention, the attention operation of each he…