1 paper · 1 filter
Hiroyasu Usami, Keisuke Hara, Ayato Tsuboi +1
Fusing independently normalized attention heads can increase the number of explicit key-value entries, or atoms, needed to preserve their output. We prove a sharp separation for $W…