6 citations · 10 across the 4 of their papers we have counts for
5 papers · 1 filter
When and Why Momentum Accelerates SGD:An Empirical Study
Jingwen Fu, Bohan Wang, Huishuai Zhang +3
Momentum has become a crucial component in deep learning optimizers, necessitating a comprehensive understanding of when and why it accelerates stochastic gradient descent (SGD). T…
Optimizing Information-theoretical Generalization Bounds via Anisotropic Noise in SGLD
Bohan Wang, Huishuai Zhang, Jieyu Zhang +3
Recently, the information-theoretical framework has been proven to be able to obtain non-vacuous generalization bounds for large models trained by Stochastic Gradient Langevin Dyna…
Robustness, Privacy, and Generalization of Adversarial Training
Fengxiang He, Shaopeng Fu, Bohan Wang +1
Adversarial training can considerably robustify deep neural networks to resist adversarial attacks. However, some works suggested that adversarial training might comprise the priva…
Tighter Generalization Bounds for Iterative Differentially Private Learning Algorithms
Fengxiang He, Bohan Wang, Dacheng Tao
This paper studies the relationship between generalization and privacy preservation in iterative learning algorithms by two sequential steps. We first establish an alignment betwee…
Piecewise linear activations substantially shape the loss surfaces of neural networks
Fengxiang He, Bohan Wang, Dacheng Tao
Understanding the loss surface of a neural network is fundamentally important to the understanding of deep learning. This paper presents how piecewise linear activation functions s…