Universal Transformers for Circuit Computations: Perfect Length Generalization in Tiny Transformers

Takuya Ito, Ruchir Puri, Murray Campbell +1

Learning generalizable algorithmic computations remains a challenge for neural networks, as reflected in persistent failures on compositional and length generalization benchmarks.…