245 citations · 682 across the 20 of their papers we have counts for
21 papers
A view of mini-batch SGD via generating functions: conditions of convergence, phase transitions, benefit from negative momenta
Maksim Velikanov, Denis Kuznedelev, Dmitry Yarotsky
Mini-batch SGD with momentum is a fundamental algorithm for learning large predictive models. In this paper we develop a new analytic framework to analyze noise-averaged properties…
Embedded Ensembles: Infinite Width Limit and Operating Regimes
Maksim Velikanov, Roman Kail, Ivan Anokhin +4
A memory efficient approach to ensembling neural networks is to share most weights among the ensembled models by means of a single reference network. We refer to this strategy as E…
Tight Convergence Rate Bounds for Optimization Under Power Law Spectral Conditions
Maksim Velikanov, Dmitry Yarotsky
Performance of optimization on quadratic problems sensitively depends on the low-lying part of the spectrum. For large (effectively infinite-dimensional) problems, this part of the…
Universal scaling laws in the gradient descent training of neural networks
Maksim Velikanov, Dmitry Yarotsky
Current theoretical results on optimization trajectories of neural networks trained by gradient descent typically have the form of rigorous but potentially loose bounds on the loss…
Elementary superexpressive activations
Dmitry Yarotsky
We call a finite family of activation functions superexpressive if any multivariate continuous function can be approximated by a neural network that uses these activations and has…
Low-loss connection of weight vectors: distribution-based approaches
Ivan Anokhin, Dmitry Yarotsky
Recent research shows that sublevel sets of the loss surfaces of overparameterized networks are connected, exactly or approximately. We describe and compare experimentally a panel…