Error Feedback Can Accurately Compress Preconditioners

Ionut-Vlad Modoranu, Aleksei Kalinov, Eldar Kurtic +2

Leveraging second-order information about the loss at the scale of deep networks is one of the main lines of approach for improving the performance of current optimizers for deep l…