Adapprox: Adaptive Approximation in Adam Optimization via Randomized Low-Rank Matrices

Pengxiang Zhao, Ping Li, Yingjie Gu +4

As deep learning models exponentially increase in size, optimizers such as Adam encounter significant memory consumption challenges due to the storage of first and second moment da…