73 citations · 142 across the 31 of their papers we have counts for
7 papers · 1 filter
Gradient Descent with Polyak's Momentum Finds Flatter Minima via Large Catapults
Prin Phunyaphibarn, Junghyun Lee, Bohan Wang +2
Although gradient descent with Polyak's momentum is widely used in modern machine and deep learning, a concrete understanding of its effects on the training trajectory remains elus…
Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexity
Bohan Wang, Jingwen Fu, Huishuai Zhang +2
Recently, Arjevani et al. [1] established a lower bound of iteration complexity for the first-order optimization under an -smooth condition and a bounded noise variance assumpti…
When and Why Momentum Accelerates SGD:An Empirical Study
Jingwen Fu, Bohan Wang, Huishuai Zhang +3
Momentum has become a crucial component in deep learning optimizers, necessitating a comprehensive understanding of when and why it accelerates stochastic gradient descent (SGD). T…
Fast Conditional Mixing of MCMC Algorithms for Non-log-concave Distributions
Xiang Cheng, Bohan Wang, Jingzhao Zhang +1
MCMC algorithms offer empirically efficient tools for sampling from a target distribution . However, on the theory side, MCMC algorithms suffer from slow…
Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptions
Bohan Wang, Huishuai Zhang, Zhi-Ming Ma +1
We provide a simple convergence proof for AdaGrad optimizing non-convex objectives under only affine noise variance and bounded smoothness assumptions. The proof is essentially bas…
On the Trade-off of Intra-/Inter-class Diversity for Supervised Pre-training
Jieyu Zhang, Bohan Wang, Zhengyu Hu +2
Pre-training datasets are critical for building state-of-the-art machine learning models, motivating rigorous study on their impact on downstream tasks. In this work, we study the…