Polygonal Unadjusted Langevin Algorithms: Creating stable and efficient adaptive algorithms for neural networks

Abstract

We present a new class of Langevin based algorithms, which overcomes many of the known shortcomings of popular adaptive optimizers that are currently used for the fine tuning of deep learning models. Its underpinning theory relies on recent advances of Euler's polygonal approximations for stochastic differential equations (SDEs) with monotone coefficients. As a result, it inherits the stability properties of tamed algorithms, while it addresses other known issues, e.g. vanishing gradients in neural networks. In particular, we provide a nonasymptotic analysis and full theoretical guarantees for the convergence properties of an algorithm of this novel class, which we named TH$\epsilon$O POULA (or, simply, TheoPouLa). Finally, several experiments are presented with different types of deep learning models, which show the superior performance of TheoPouLa over many popular adaptive optimization algorithms.

References in corpus (12)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Decoupled Weight Decay Regularization
• On the difficulty of training Recurrent Neural Networks
• On the Convergence of Adam and Beyond
• Strong convergence of an explicit numerical method for SDEs with nonglobally Lipschitz continuous coefficients
• Adding Gradient Noise Improves Learning for Very Deep Networks
• A note on tamed Euler approximations
• Non-asymptotic convergence analysis for the Unadjusted Langevin Algorithm
• The promises and pitfalls of Stochastic Gradient Langevin Dynamics
• On stochastic gradient Langevin dynamics with dependent data streams: the fully non-convex case
• Higher Order Langevin Monte Carlo Algorithm
• Nonasymptotic estimates for Stochastic Gradient Langevin Dynamics under local conditions in nonconvex optimization