Adaptive Step Sizes in Variance Reduction via Regularization

Abstract

The main goal of this work is equipping convex and nonconvex problems with Barzilai-Borwein (BB) step size. With the adaptivity of BB step sizes granted, they can fail when the objective function is not strongly convex. To overcome this challenge, the key idea here is to bridge (non)convex problems and strongly convex ones via regularization. The proposed regularization schemes are \textit{simple} yet effective. Wedding the BB step size with a variance reduction method, known as SARAH, offers a free lunch compared with vanilla SARAH in convex problems. The convergence of BB step sizes in nonconvex problems is also established and its complexity is no worse than other adaptive step sizes such as AdaGrad. As a byproduct, our regularized SARAH methods for convex functions ensure that the complexity to find $\mathbb{E}[\Vert \nabla f(\mathbf{x}){\Vert }^2]\le \epsilon$ is $\mathcal{O}((n+\frac{1}{\sqrt{\epsilon }})\ln \frac{1}{\epsilon })$, improving $\epsilon$ dependence over existing results. Numerical tests further validate the merits of proposed approaches.

References in corpus (5)

• On the Convergence of Adam and Beyond
• A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets
• Estimate Sequences for Variance-Reduced Stochastic Composite Optimization
• The Complexity of Making the Gradient Small in Stochastic Convex Optimization
• Accelerating Mini-batch SARAH by Step Size Rules