LightMTP: Lightweight Latent Multi-Token Prediction

Tamara Czinczoll, Julie Kallini, Gerard de Melo +1

Next-token prediction (NTP) is the standard pretraining objective for large language models, yet it provides an explicit training signal only for the immediate next token, which ca…