On The Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learning

Thomas T. Zhang, Behrad Moniri, Ansh Nagwekar +4

Layer-wise preconditioning methods are a family of memory-efficient optimization algorithms that introduce preconditioners per axis of each layer's weight tensors. These methods ha…