What Makes Looped Transformers Perform Better Than Non-Recursive Ones

Zixuan Gong, Yong Liu, Jiaye Teng

While looped transformers (termed as Looped-Attn) often outperform standard transformers (termed as Single-Attn) on complex reasoning tasks, the mechanism for this advantage remain…