UpLoop: Upcycling Large Language Models with Gated Loop

Tieyuan Chen, Huabin Liu, Yingtao Zhang +4

Looped large language models trade parameter storage for recurrent computation, but most are trained from scratch. In practice, pretraining with looped depth is expensive, making p…