Implicit Gradient Regularization

David G. T. Barrett, Benoit Dherin

Gradient descent can be surprisingly good at optimizing deep neural networks without overfitting and without explicit regularization. We find that the discrete steps of gradient de…