17 citations · 27 across the 8 of their papers we have counts for
Showing 2021Show all
3 papers · 1 filter
cs.LG2021
Provable Convergence of Nesterov's Accelerated Gradient Method for Over-Parameterized Neural Networks
Xin Liu, Zhisong Pan, Wei Tao
Momentum methods, such as heavy ball method~(HB) and Nesterov's accelerated gradient method~(NAG), have been widely used in training neural networks by incorporating the history of…
cs.LG2021★ 6 cited
The Role of Momentum Parameters in the Optimal Convergence of Adaptive Polyak's Heavy-ball Methods
Wei Tao, Sheng Long, Gaowei Wu +1
The adaptive stochastic gradient descent (SGD) with momentum has been widely adopted in deep learning as well as convex optimization. In practice, the last iterate is commonly used…
cs.LG2021
Gradient Descent Averaging and Primal-dual Averaging for Strongly Convex Optimization
Wei Tao, Wei Li, Zhisong Pan +1
Averaging scheme has attracted extensive attention in deep learning as well as traditional machine learning. It achieves theoretically optimal convergence and also improves the emp…