2 papers
cs.LG2026
Why Adam Can Beat SGD: Second-Moment Normalization Yields Sharper Tails
Ruinan Jin, Yingbin Liang, Shaofeng Zou
Despite Adam demonstrating faster empirical convergence than SGD in many applications, much of the existing theory yields guarantees essentially comparable to those of SGD, leaving…
cs.LG2024
Neural Networks with Sparse Activation Induced by Large Bias: Tighter Analysis with Bias-Generalized NTK
Hongru Yang, Ziyu Jiang, Ruizhe Zhang +2
We study training one-hidden-layer ReLU networks in the neural tangent kernel (NTK) regime, where the networks' biases are initialized to some constant rather than zero. We prove t…