82 citations · 546 across the 65 of their papers we have counts for
8 papers · 1 filter
Cubic Regularization with Momentum for Nonconvex Optimization
Zhe Wang, Yi Zhou, Yingbin Liang +1
Momentum is a popular technique to accelerate the convergence in practical training, and its impact on convergence guarantee has been well-studied for first-order algorithms. Howev…
Optimal Adaptive and Accelerated Stochastic Gradient Descent
Qi Deng, Yi Cheng, Guanghui Lan
Stochastic gradient descent (\textsc{Sgd}) methods are the most powerful optimization tools in training machine learning and deep learning models. Moreover, acceleration (a.k.a. mo…
Asynchronous decentralized accelerated stochastic gradient descent
Guanghui Lan, Yi Zhou
In this work, we introduce an asynchronous decentralized accelerated stochastic gradient descent type of method for decentralized stochastic optimization, considering communication…
Complexity of Training ReLU Neural Network
Digvijay Boob, Santanu S. Dey, Guanghui Lan
In this paper, we explore some basic questions on the complexity of training neural networks with ReLU activation function. We show that it is NP-hard to train a two-hidden layer f…
A Note on Inexact Condition for Cubic Regularized Newton's Method
Zhe Wang, Yi Zhou, Yingbin Liang +1
This note considers the inexact cubic-regularized Newton's method (CR), which has been shown in \cite{Cartis2011a} to achieve the same order-level convergence rate to a secondary s…
Accelerated Stochastic Algorithms for Nonconvex Finite-sum and Multi-block Optimization
Guanghui Lan, Yu Yang
In this paper, we present new stochastic methods for solving two important classes of nonconvex optimization problems. We first introduce a randomized accelerated proximal gradient…