Thinking into the Future: Latent Lookahead Training for Transformers

Lorenzo Noci, Gregor Bachmann, Seyed-Mohsen Moosavi-Dezfooli +1

Autoregressive language models trained with next-token prediction generate text by sampling one discrete token at a time. Although very scalable, this objective forces the model to…