2 citations · 3 across the 14 of their papers we have counts for
4 papers · 1 filter
Convergence rates for the RMSprop optimizer with full control of the hyperparameters
Steffen Dereich, Arnulf Jentzen
Popular adaptive stochastic gradient descent (SGD) methods to train artificial intelligence (AI) systems include the RMSprop, the Adam, and the AdamW optimizers, where the adaptivi…
On MUON optimization: From non-convergence to an error analysis with Polar Express and the Newton-Schulz polynomial from implementations
Thang Do, Steffen Dereich, Arnulf Jentzen
Stochastic gradient descent (SGD) optimization methods are the standard instruments for the training of deep neural networks (DNNs). In many relevant artificial intelligence (AI) s…
Central limit theorem for the averaged Adam optimizer
Steffen Dereich, Arnulf Jentzen
In this article, we analyse convergence of the averaged Adam optimizer to an attracting zero of the Adam vector field. We provide a central limit theorem that, in particular, quant…
Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method
Steffen Dereich, Thang Do, Arnulf Jentzen
The adaptive moment estimation (Adam) optimizer proposed by Kingma & Ba (2014) is presumably the most popular stochastic gradient descent (SGD) optimization method for the training…