How Transformers Learn to Plan via Multi-Token Prediction

Jianhao Huang, Zhanpeng Zhou, Renqiu Xia +3

While next-token prediction (NTP) has been the standard objective for training language models, it often struggles to capture global structure in reasoning tasks. Multi-token predi…