8 papers
Learning rate adaptive stochastic gradient descent optimization methods: numerical simulations for deep learning methods for partial differential equations and convergence analyses
Steffen Dereich, Arnulf Jentzen, Adrian Riekert
The standard stochastic gradient descent (SGD) optimization method, as well as adaptive methods such as the Adam optimizer fail to converge if the learning rates do not converge to…
Algorithmically Designed Artificial Neural Networks (ADANNs): Higher order deep operator learning for parametric partial differential equations
Arnulf Jentzen, Adrian Riekert, Philippe von Wurstemberger
In this article we propose a new deep learning approach to approximate operators related to parametric partial differential equations (PDEs). In particular, we introduce a new stra…
Convergence to good non-optimal critical points in the training of neural networks: Gradient descent optimization with one random initialization overcomes all bad non-global local minima with high probability
Shokhrukh Ibragimov, Arnulf Jentzen, Adrian Riekert
Gradient descent (GD) methods for the training of artificial neural networks (ANNs) belong nowadays to the most heavily employed computational schemes in the digital world. Despite…
Asymptotic stability properties and a priori bounds for Adam and other gradient descent optimization methods
Steffen Dereich, Robin Graeber, Arnulf Jentzen +1
Gradient descent (GD) based optimization methods are these days the standard tools to train deep neural networks in artificial intelligence systems. In optimization procedures in d…
PADAM: Parallel averaged Adam reduces the error for stochastic optimization in scientific machine learning
Arnulf Jentzen, Julian Kranz, Adrian Riekert
Averaging techniques such as Ruppert--Polyak averaging and exponential movering averaging (EMA) are powerful approaches to accelerate optimization procedures of stochastic gradient…
Sharp higher order convergence rates for the Adam optimizer
Steffen Dereich, Arnulf Jentzen, Adrian Riekert
Gradient descent based optimization methods are the methods of choice to train deep neural networks in machine learning. Beyond the standard gradient descent method, also suitable…