2 citations · 3 across the 14 of their papers we have counts for
15 papers
Convergence rates for the RMSprop optimizer with full control of the hyperparameters
Steffen Dereich, Arnulf Jentzen
Popular adaptive stochastic gradient descent (SGD) methods to train artificial intelligence (AI) systems include the RMSprop, the Adam, and the AdamW optimizers, where the adaptivi…
On MUON optimization: From non-convergence to an error analysis with Polar Express and the Newton-Schulz polynomial from implementations
Thang Do, Steffen Dereich, Arnulf Jentzen
Stochastic gradient descent (SGD) optimization methods are the standard instruments for the training of deep neural networks (DNNs). In many relevant artificial intelligence (AI) s…
Central limit theorem for the averaged Adam optimizer
Steffen Dereich, Arnulf Jentzen
In this article, we analyse convergence of the averaged Adam optimizer to an attracting zero of the Adam vector field. We provide a central limit theorem that, in particular, quant…
Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method
Steffen Dereich, Thang Do, Arnulf Jentzen
The adaptive moment estimation (Adam) optimizer proposed by Kingma & Ba (2014) is presumably the most popular stochastic gradient descent (SGD) optimization method for the training…
ODE approximation for the Adam algorithm: General and overparametrized setting
Steffen Dereich, Arnulf Jentzen, Sebastian Kassing
The Adam optimizer is currently presumably the most popular optimization method in deep learning. In this article we develop an ODE based method to study the Adam optimizer in a fa…
Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer
Steffen Dereich, Thang Do, Arnulf Jentzen +1
Beside the standard stochastic gradient descent (SGD) method, the Adam optimizer due to Kingma & Ba (2014) is currently probably the best-known optimization method for the training…
Asymptotic stability properties and a priori bounds for Adam and other gradient descent optimization methods
Steffen Dereich, Robin Graeber, Arnulf Jentzen +1
Gradient descent (GD) based optimization methods are these days the standard tools to train deep neural networks in artificial intelligence systems. In optimization procedures in d…
SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures
Julian Kranz, Davide Gallon, Steffen Dereich +1
We study gradient flows for loss landscapes of fully connected feedforward neural networks with commonly used continuously differentiable activation functions such as the logistic,…
Sharp higher order convergence rates for the Adam optimizer
Steffen Dereich, Arnulf Jentzen, Adrian Riekert
Gradient descent based optimization methods are the methods of choice to train deep neural networks in machine learning. Beyond the standard gradient descent method, also suitable…
In almost all shallow analytic neural network optimization landscapes, efficient minimizers have strongly convex neighborhoods
Felix Benning, Steffen Dereich
Whether or not a local minimum of a cost function has a strongly convex neighborhood greatly influences the asymptotic convergence rate of optimizers. In this article, we rigorousl…
Mathematical analysis of the gradients in deep learning
Steffen Dereich, Thang Do, Arnulf Jentzen +1
Deep learning algorithms -- typically consisting of a class of deep artificial neural networks (ANNs) trained by a stochastic gradient descent (SGD) optimization method -- are nowa…
Averaged Adam accelerates stochastic optimization in the training of deep neural network approximations for partial differential equation and optimal control problems
Steffen Dereich, Arnulf Jentzen, Adrian Riekert
Deep learning methods - usually consisting of a class of deep neural networks (DNNs) trained by a stochastic gradient descent (SGD) optimization method - are nowadays omnipresent i…
Convergence rates for the Adam optimizer
Steffen Dereich, Arnulf Jentzen
Stochastic gradient descent (SGD) optimization methods are nowadays the method of choice for the training of deep neural networks (DNNs) in artificial intelligence systems. In prac…
Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning rates
Steffen Dereich, Robin Graeber, Arnulf Jentzen
Deep learning algorithms - typically consisting of a class of deep neural networks trained by a stochastic gradient descent (SGD) optimization method - are nowadays the key ingredi…
Learning rate adaptive stochastic gradient descent optimization methods: numerical simulations for deep learning methods for partial differential equations and convergence analyses
Steffen Dereich, Arnulf Jentzen, Adrian Riekert
The standard stochastic gradient descent (SGD) optimization method, as well as adaptive methods such as the Adam optimizer fail to converge if the learning rates do not converge to…