Length Generalization with Log-Depth Recurrent Units

Charles Pert, Dalal Alrajeh, Alessandra Russo

Length generalization remains a persistent challenge for neural networks: recurrent models tend to suffer from positional biases, while transformers are constrained by fixed comput…