14 citations · 19 across the 5 of their papers we have counts for
5 papers · 1 filter
The Marginal Value of Momentum for Small Learning Rate SGD
Runzhe Wang, Sadhika Malladi, Tianhao Wang +2
Momentum is known to accelerate the convergence of gradient descent in strongly convex settings without stochastic gradient noise. In stochastic optimization, such as training neur…
Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Bias
Kaifeng Lyu, Zhiyuan Li, Runzhe Wang +1
The generalization mystery of overparametrized deep nets has motivated efforts to understand how gradient descent (GD) converges to low-loss solutions that generalize well. Real-li…
Optimal Gradient-based Algorithms for Non-concave Bandit Optimization
Baihe Huang, Kaixuan Huang, Sham M. Kakade +4
Bandit problems with linear or concave reward have been extensively studied, but relatively few works have studied bandits with non-concave reward. This work considers a large fami…
Going Beyond Linear RL: Sample Efficient Neural Function Approximation
Baihe Huang, Kaixuan Huang, Sham M. Kakade +4
Deep Reinforcement Learning (RL) powered by neural net approximation of the Q function has had enormous empirical success. While the theory of RL has traditionally focused on linea…
Mildly Overparametrized Neural Nets can Memorize Training Data Efficiently
Rong Ge, Runzhe Wang, Haoyu Zhao
It has been observed \citep{zhang2016understanding} that deep neural networks can memorize: they achieve 100\% accuracy on training data. Recent theoretical results explained such…