1.8k citations · 7.5k across the 146 of their papers we have counts for
152 papers
Convergence rates for the RMSprop optimizer with full control of the hyperparameters
Steffen Dereich, Arnulf Jentzen
Popular adaptive stochastic gradient descent (SGD) methods to train artificial intelligence (AI) systems include the RMSprop, the Adam, and the AdamW optimizers, where the adaptivi…
On MUON optimization: From non-convergence to an error analysis with Polar Express and the Newton-Schulz polynomial from implementations
Thang Do, Steffen Dereich, Arnulf Jentzen
Stochastic gradient descent (SGD) optimization methods are the standard instruments for the training of deep neural networks (DNNs). In many relevant artificial intelligence (AI) s…
Strong error analysis for the stochastic momentum optimizer
Davide Gallon, Arnulf Jentzen
Stochastic gradient descent (SGD) optimization schemes are the methods of choice for the optimization of deep neural networks (DNNs) in artificial intelligence (AI) systems. Often…
Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks
Shokhrukh Ibragimov, Arnulf Jentzen
Gradient based optimization methods are nowadays the methods of choice for training deep neural networks (DNNs) in artificial intelligence (AI) systems. In practically relevant DNN…
Central limit theorem for the averaged Adam optimizer
Steffen Dereich, Arnulf Jentzen
In this article, we analyse convergence of the averaged Adam optimizer to an attracting zero of the Adam vector field. We provide a central limit theorem that, in particular, quant…
Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method
Steffen Dereich, Thang Do, Arnulf Jentzen
The adaptive moment estimation (Adam) optimizer proposed by Kingma & Ba (2014) is presumably the most popular stochastic gradient descent (SGD) optimization method for the training…