When Can Attention Heads Be Statically Defined?

Weixian Waylon Li, Yintao Tai, Marcio Fonseca +1

Some attention heads learn similar patterns across inputs. Reusing these patterns could reduce training cost by avoiding repeated query-key score computation and softmax. Through c…