Selective Attention Improves Transformer

Yaniv Leviathan, Matan Kalman, Yossi Matias

Unneeded elements in the attention's context degrade performance. We introduce Selective Attention, a simple parameter-free change to the standard attention mechanism which reduces…