NDP: Next Distribution Prediction as a More Broad Target

Junhao Ruan, Abudukeyumu Abudula, Xinyu Liu +7

Large language models (LLMs) trained on next-token prediction (NTP) paradigm have demonstrated powerful capabilities. However, the existing NTP paradigm contains several limitation…