212 citations · 575 across the 20 of their papers we have counts for
6 papers · 1 filter
Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
Difan Zou, Yuan Cao, Dongruo Zhou +1
We study the problem of training deep neural networks with Rectified Linear Unit (ReLU) activation function using gradient descent and stochastic gradient descent. In particular, w…
Sample Efficient Stochastic Variance-Reduced Cubic Regularization Method
Dongruo Zhou, Pan Xu, Quanquan Gu
We propose a sample efficient stochastic variance-reduced cubic regularization (Lite-SVRC) algorithm for finding the local minimum efficiently in nonconvex optimization. The propos…
On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization
Dongruo Zhou, Jinghui Chen, Yuan Cao +2
Adaptive gradient methods are workhorses in deep learning. However, the convergence guarantees of adaptive gradient methods for nonconvex optimization have not been thoroughly stud…
Finding Local Minima via Stochastic Nested Variance Reduction
Dongruo Zhou, Pan Xu, Quanquan Gu
We propose two algorithms that can find local minima faster than the state-of-the-art algorithms in both finite-sum and general stochastic nonconvex optimization. At the core of th…
Stochastic Nested Variance Reduction for Nonconvex Optimization
Dongruo Zhou, Pan Xu, Quanquan Gu
We study finite-sum nonconvex optimization problems, where the objective function is an average of nonconvex functions. We propose a new stochastic gradient descent algorithm b…
Stochastic Variance-Reduced Cubic Regularized Newton Method
Dongruo Zhou, Pan Xu, Quanquan Gu
We propose a stochastic variance-reduced cubic regularized Newton method for non-convex optimization. At the core of our algorithm is a novel semi-stochastic gradient along with a…