Transformers Can Navigate Mazes With Multi-Step Prediction

Niklas Nolte, Ouail Kitouni, Adina Williams +2

Despite their remarkable success in language modeling, transformers trained to predict the next token in a sequence struggle with long-term planning. This limitation is particularl…