306 citations · 532 across the 22 of their papers we have counts for
23 papers
When and Why Momentum Accelerates SGD:An Empirical Study
Jingwen Fu, Bohan Wang, Huishuai Zhang +3
Momentum has become a crucial component in deep learning optimizers, necessitating a comprehensive understanding of when and why it accelerates stochastic gradient descent (SGD). T…
Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptions
Bohan Wang, Huishuai Zhang, Zhi-Ming Ma +1
We provide a simple convergence proof for AdaGrad optimizing non-convex objectives under only affine noise variance and bounded smoothness assumptions. The proof is essentially bas…
Provable Adaptivity of Adam under Non-uniform Smoothness
Bohan Wang, Yushun Zhang, Huishuai Zhang +6
Adam is widely adopted in practical applications due to its fast convergence. However, its theoretical analysis is still far from satisfactory. Existing convergence analyses for Ad…
Normalized/Clipped SGD with Perturbation for Differentially Private Non-Convex Optimization
Xiaodong Yang, Huishuai Zhang, Wei Chen +1
By ensuring differential privacy in the learning algorithms, one can rigorously mitigate the risk of large models memorizing sensitive training data. In this paper, we study two al…
Neural Operator with Regularity Structure for Modeling Dynamics Driven by SPDEs
Peiyan Hu, Qi Meng, Bingguang Chen +6
Stochastic partial differential equations (SPDEs) are significant tools for modeling dynamics in many areas including atmospheric sciences and physics. Neural Operators, generation…
Optimizing Information-theoretical Generalization Bounds via Anisotropic Noise in SGLD
Bohan Wang, Huishuai Zhang, Jieyu Zhang +3
Recently, the information-theoretical framework has been proven to be able to obtain non-vacuous generalization bounds for large models trained by Stochastic Gradient Langevin Dyna…