On the "Induction Bias" in Sequence Models

M. Reza Ebrahimi, Michaël Defferrard, Sunny Panchal +1

Despite the remarkable practical success of transformer-based language models, recent work has raised concerns about their ability to perform state tracking. In particular, a growi…