Limitations of neural network training due to numerical instability of backpropagation

Clemens Karner, Vladimir Kazeev, Philipp Christian Petersen

We study the training of deep neural networks by gradient descent where floating-point arithmetic is used to compute the gradients. In this framework and under realistic assumption…