TRA: Better Length Generalisation with Threshold Relative Attention

Mattia Opper, Roland Fernandez, Paul Smolensky +1

Transformers struggle with length generalisation, displaying poor performance even on basic tasks. We test whether these limitations can be explained through two key failures of th…