Faster Stochastic Quasi-Newton Methods

Abstract

Stochastic optimization methods have become a class of popular optimization tools in machine learning. Especially, stochastic gradient descent (SGD) has been widely used for machine learning problems such as training neural networks due to low per-iteration computational complexity. In fact, the Newton or quasi-newton methods leveraging second-order information are able to achieve a better solution than the first-order methods. Thus, stochastic quasi-Newton (SQN) methods have been developed to achieve the better solution efficiently than the stochastic first-order methods by utilizing approximate second-order information. However, the existing SQN methods still do not reach the best known stochastic first-order oracle (SFO) complexity. To fill this gap, we propose a novel faster stochastic quasi-Newton method (SpiderSQN) based on the variance reduced technique of SIPDER. We prove that our SpiderSQN method reaches the best known SFO complexity of $\mathcal{O}(n+n^{1/2}{\epsilon }^{-2})$ in the finite-sum setting to obtain an $\epsilon$-first-order stationary point. To further improve its practical performance, we incorporate SpiderSQN with different momentum schemes. Moreover, the proposed algorithms are generalized to the online setting, and the corresponding SFO complexity of $\mathcal{O}({\epsilon }^{-3})$ is developed, which also matches the existing best result. Extensive experiments on benchmark datasets demonstrate that our new algorithms outperform state-of-the-art approaches for nonconvex optimization.

arXiv comment11 pages, accepted for publication by TNNLS. arXiv admin note: text overlap with arXiv:1902.02715 by other authors

References in corpus (13)

• SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimator
• A Linearly-Convergent Stochastic L-BFGS Algorithm
• Non-convex Finite-Sum Optimization Via SCSG Methods
• Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods
• Stochastic Recursive Gradient Algorithm for Nonconvex Optimization
• Stochastic Nested Variance Reduction for Nonconvex Optimization
• Convergence Analysis of Proximal Gradient with Momentum for Nonconvex Optimization
• A Variance Reduced Stochastic Newton Method
• Faster Stochastic Alternating Direction Method of Multipliers for Nonconvex Optimization
• Learning with Non-Convex Truncated Losses by SGD
• Stochastic Recursive Variance-Reduced Cubic Regularization Methods
• Momentum Schemes with Stochastic Variance Reduction for Nonconvex Composite Optimization
• Cubic Regularization with Momentum for Nonconvex Optimization