Non-Singularity of the Gradient Descent map for Neural Networks with Piecewise Analytic Activations

Alexandru Crăciun, Debarghya Ghoshdastidar

The theory of training deep networks has become a central question of modern machine learning and has inspired many practical advancements. In particular, the gradient descent (GD)…