Hierarchical Latent Prediction for Language Models

Chang Shi, Tim Pearce, Manan Tomar +2

While standard Next-Token Prediction (NTP) lays the foundation of language model pre- training, its teacher-forced training paradigm may not be optimal for long-horizon reasoning a…