Untangling tradeoffs between recurrence and self-attention in neural networks

Giancarlo Kerg, Bhargav Kanuparthi, Anirudh Goyal +3

Attention and self-attention mechanisms, are now central to state-of-the-art deep learning on sequential tasks. However, most recent progress hinges on heuristic approaches with li…