Putting Transformer Theories to the Test

Awni Altabaa, Tobias Schnabel, Kiran Tomlinson

Several formal theories have been developed to characterize the capabilities and limitations of the transformer architecture, each employing different abstractions that capture dif…