Think before you speak: Training Language Models With Pause Tokens

Sachin Goyal, Ziwei Ji, Ankit Singh Rawat +3

Language models generate responses by producing a series of tokens in immediate succession: the $(K+1{)}^{th}$ token is an outcome of manipulating $K$ hidden vectors per layer, one v…