Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts

Di He, Pengxiang Li, Da Chang +3

Looped Transformers introduce recurrent depth as a new scaling axis for LLMs: by repeatedly applying shared Transformer blocks, they increase effective depth without increasing par…