Two-Point Deterministic Equivalence for Stochastic Gradient Dynamics in Linear Models

Abstract

We derive a novel deterministic equivalence for the two-point function of a random matrix resolvent. Using this result, we give a unified derivation of the performance of a wide variety of high-dimensional linear models trained with stochastic gradient descent. This includes high-dimensional linear regression, kernel regression, and linear random feature models. Our results include previously known asymptotics as well as novel ones.

22 pages, in press at Advances in Theoretical and Mathematical Physics