Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization

Sharan Sahu, Abir Sarkar, Cameron J. Hogan +1

We provide a theoretical analysis of Adam under non-stationary stochastic objectives, separating two regimes: Euclidean tracking under adaptive strong monotonicity of the Adam-prec…