Looped Transformers with Layer Normalization Provably Learn the Power Method

Lyumin Wu, Chenyang Zhang, Yuan Cao

Transformers have achieved remarkable success across a wide range of applications, and a growing body of work suggests that part of their strength comes from their ability to learn…