LT2: Linear-Time Looped Transformers

Chunyuan Deng, Yizhe Zhang, Rui-Jie Zhu +4

Looped Transformers (LT) have emerged as a powerful architecture by iterating their layers multiple times before decoding the final token. However, pairing them with full attention…