Adaptive norms for deep learning with regularized Newton methods

Abstract

We investigate the use of regularized Newton methods with adaptive norms for optimizing neural networks. This approach can be seen as a second-order counterpart of adaptive gradient methods, which we here show to be interpretable as first-order trust region methods with ellipsoidal constraints. In particular, we prove that the preconditioning matrix used in RMSProp and Adam satisfies the necessary conditions for provable convergence of second-order trust region methods with standard worst-case complexities on general non-convex objectives. Furthermore, we run experiments across different neural architectures and datasets to find that the ellipsoidal constraints constantly outperform their spherical counterpart both in terms of number of backpropagations and asymptotic loss value. Finally, we find comparable performance to state-of-the-art first-order methods in terms of backpropagations, but further advances in hardware are needed to render Newton methods competitive in terms of computational time.

References in corpus (10)

• A Convergence Theory for Deep Learning via Over-Parameterization
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Revisiting Small Batch Training for Deep Neural Networks
• New insights and perspectives on the natural gradient method
• Three Factors Influencing Minima in SGD
• On the Power of Over-parametrization in Neural Networks with Quadratic Activation
• Escaping Saddles with Stochastic Gradients
• Failures of Gradient-Based Deep Learning
• Inexact Non-Convex Newton-Type Methods
• Stochastic Second-order Methods for Non-convex Optimization with Inexact Hessian and Gradient