How to Loop MoE: Flatten the Experts, Untie the Attention

Shouren Wang, Chuang Ma, Mohsen Hariri +6

Looped Transformers reuse one block of layers several times: by spending extra computation they push a model of fixed size further, and so use its parameters more fully; while spar…