Understanding SOAP from the Perspective of Gradient Whitening

Yanqing Lu, Letao Wang, Jinbo Liu

Shampoo with Adam in the Preconditioner's eigenbasis (SOAP) has recently emerged as a promising optimization algorithm for neural network training, achieving superior training effi…