Delayed Attention Training Improves Length Generalization in Transformer--RNN Hybrids

Buu Phan, Reza Ebrahimi, Sanjay Haresh +1

We study length generalization in sequence models on a composite problem involving both state tracking and associative recall. Prior work finds that recurrent networks handle state…