Extreme Tensoring for Low-Memory Preconditioning

Xinyi Chen, Naman Agarwal, Elad Hazan +2

State-of-the-art models are now trained with billions of parameters, reaching hardware limits in terms of memory consumption. This has created a recent demand for memory-efficient…