Provable Long-Range Benefits of Next-Token Prediction

Xinyuan Cao, Santosh S. Vempala

Why do modern language models, trained to do well on next-word prediction, appear to generate coherent documents and capture long-range structure? Here we show that next-token pred…