NS-ATTENTION: Newton-Schulz Transformations of Attention Outputs in Vision Transformers

Xiaohe Jiang, Guoqiang Zhang, Tianjin Huang +1

Newton-Schulz (NS) iteration has recently been used in the Muon optimizer to transform update matrices during the training of large language models. Motivated by its spectral effec…