Multi-Token Prediction Needs Registers

Anastasios Gerontopoulos, Spyros Gidaris, Nikos Komodakis

Multi-token prediction has emerged as a promising objective for improving language model pretraining, but its benefits have not consistently generalized to other settings such as f…