Why Clipping Matters in AdaGrad? Toward a High-Probability Theory under Generalized Smoothness

Abstract

We analyze the original same-step coordinate-wise AdaGrad under generalized smoothness and heavy-tailed noise with bounded variance. In this setting, local curvature may grow sub-quadratically with the gradient norm, and stochastic gradients are assumed to have only bounded conditional second moments. We show that unclipped AdaGrad can become \emph{anisotropically miscalibrated}: under heavy-tailed noise, the adaptive denominator can learn the geometry of rare noise shocks rather than the local curvature of the objective, leading to a persistent directional distortion that blocks finite-horizon Euclidean progress. We then prove that clipping repairs this failure mode. Our main result is a finite-horizon high-probability guarantee for the original non-lagged AdaGrad update, yielding $\frac{1}{T}{\sum }_{t=0}^{T-1}\Vert \nabla f(x_t){\Vert }^2=\mathcal{O}\left(\frac{d(\sqrt{\log T}+\log \frac{1}{\delta })}{\sqrt{T}}\right),$ and hence $\tilde{\mathcal{O}}({\epsilon }^{-2})$ complexity. This shows that, for AdaGrad under heavy-tailed noise, clipping is a structural stabilizer of the adaptive geometry rather than merely a robustness heuristic.