Next Concept Prediction in Discrete Latent Space Leads to Stronger Language Models

Yuliang Liu, Yunchong Song, Yixuan Wang +6

We propose Next Concept Prediction (NCP), a generative pretraining paradigm built on top of Next Token Prediction (NTP). NCP predicts discrete concepts that span multiple tokens, t…