Semformer: Transformer Language Models with Semantic Planning

Yongjing Yin, Junran Ding, Kai Song +1

Next-token prediction serves as the dominant component in current neural language models. During the training phase, the model employs teacher forcing, which predicts tokens based…