Stochastic optimization on matrices and a graphon McKean-Vlasov limit

Abstract

We consider stochastic gradient descents on the space of large symmetric matrices of suitable functions that are invariant under permuting the rows and columns using the same permutation. We establish deterministic limits of these random curves as the dimensions of the matrices go to infinity while the entries remain bounded. Under a ``small noise'' assumption the limit is shown to be the gradient flow of functions on graphons whose existence was established in Oh, Somani, Pal, and Tripathi, \texit{J Theor Probab 37, 1469--1522 (2024)}. We also consider limits of stochastic gradient descents with added properly scaled reflected Brownian noise. The limiting curve of graphons is characterized by a family of stochastic differential equations with reflections and can be thought of as an extension of the classical McKean-Vlasov limit for interacting diffusions to the graphon setting. The proofs introduce a family of infinite-dimensional exchangeable arrays of reflected diffusions and a novel notion of propagation of chaos for large matrices of diffusions converging to such arrays in a suitable sense.

37 pages+ references. Accepted version in Annals of Applied Probability

References in corpus (8)

• An explicit formula for the Skorokhod map on $[0,a]$
• On exchangeable random variables and the statistics of large graphs and hypergraphs
• Propagation of chaos: a review of models, methods and applications. II. Applications
• A note on dynamical models on random graphs and Fokker-Planck equations
• Interacting diffusions on random graphs with diverging degrees: hydrodynamics and large deviations
• Weakly interacting oscillators on dense random graphs
• A note on Fokker-Planck equations and graphons
• Gradient flows on graphons: existence, convergence, continuity equations