Decoupling Variance and Scale-Invariant Updates in Adaptive Gradient Descent for Unified Vector and Matrix Optimization

Zitao Song, Cedar Site Bai, Zhe Zhang +2

Adaptive methods like Adam have become the $\text{de facto}$ standard for large-scale vector and Euclidean optimization due to their coordinate-wise adaptation with a second-orde…