5 papers
Gradient Flow Through Diagram Expansions: Learning Regimes and Explicit Solutions
Dmitry Yarotsky, Eugene Golikov, Yaroslav Gusev
We develop a general mathematical framework to analyze scaling regimes and derive explicit analytic solutions for gradient flow (GF) in large learning problems. Our key innovation…
A prism hierarchy of learning regimes in large linear autoencoders
Eugene Golikov, Yaroslav Gusev, Dmitry Yarotsky
Theoretical studies of machine learning models commonly consider different limiting regimes in which the learning dynamics of gradient descent becomes theoretically tractable. It i…
Corner Gradient Descent
Dmitry Yarotsky
We consider SGD-type optimization on infinite-dimensional quadratic problems with power law spectral conditions. It is well-known that on such problems deterministic GD has loss co…
SGD with memory: fundamental properties and stochastic acceleration
Dmitry Yarotsky, Maksim Velikanov
An important open problem is the theoretically feasible acceleration of mini-batch SGD-type algorithms on quadratic problems with power-law spectrum. In the non-stochastic setting,…
Learnability of high-dimensional targets by two-parameter models and gradient flow
Dmitry Yarotsky
We explore the theoretical possibility of learning -dimensional targets with -parameter models by gradient flow (GF) when . Our main result shows that if the targets are…