Sparse Layers are Critical to Scaling Looped Language Models

Ryan Lee, Jacob Biloki, Edward J. Hu +1

Looped language models repeat a set of transformer layers through depth, reducing memory costs and providing natural early-exit points at loop boundaries. However, looped models do…